on getting a barebones webapp running
Robert Koberg
I have a duplicate of the hellolift webapp running with the 2.6.1 scala
release in the latest released Jetty and Tomcat.
I about to look into this next, but one thing that would be helpful is a
brief description of why lift uses certain jars (I am a newbie with
scala code):
- httpunit
- rabbitmq-client
- smack & smackx
A couple of quick notes:
* you probably know this, but lift does not run with scala 2.7x RC.
Apparently the scala library has moved their StringBuilder from compat
to the scala root.
* In tomcat everything runs fine, but I get an error on shutdown. Here
is the full log from startup, hit the docroot index.html, and shutdown:
Feb 12, 2008 3:40:19 PM org.apache.catalina.core.ApplicationContext log
INFO: ContextListener: contextInitialized()
Feb 12, 2008 3:40:19 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: contextInitialized()
Feb 12, 2008 3:41:47 PM org.apache.catalina.session.StandardSession
writeObject
WARNING: Cannot serialize session attribute the_actor for session
D50F4D1F5232873DED227C7D1A980D48
java.io.NotSerializableException: scala.PartialFunction$$anon$1
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1156)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at
org.apache.catalina.session.StandardSession.writeObject(StandardSession.java:1515)
at
org.apache.catalina.session.StandardSession.writeObjectData(StandardSession.java:959)
at
org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:517)
at
org.apache.catalina.session.StandardManager.unload(StandardManager.java:463)
at
org.apache.catalina.session.StandardManager.stop(StandardManager.java:667)
at
org.apache.catalina.core.StandardContext.stop(StandardContext.java:4512)
at
org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:924)
at
org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1189)
at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1160)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:313)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1086)
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098)
at
org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:448)
at
org.apache.catalina.core.StandardService.stop(StandardService.java:584)
at
org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:628)
at org.apache.catalina.startup.Catalina.start(Catalina.java:603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Feb 12, 2008 3:41:47 PM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: contextDestroyed()
Feb 12, 2008 3:41:47 PM org.apache.catalina.core.ApplicationContext log
INFO: ContextListener: contextDestroyed()
David Bernard
Robert Koberg wrote:
> Hi,
>
> I have a duplicate of the hellolift webapp running with the 2.6.1 scala
> release in the latest released Jetty and Tomcat.
>
> I about to look into this next, but one thing that would be helpful is a
> brief description of why lift uses certain jars (I am a newbie with
> scala code):
IIRC :
> - httpunit
is a transitive dependency from jwebunit => test
> - rabbitmq-client
is for amqp (a queue/message protocol )
IMHO : should be optional
> - smack & smackx
is for xmpp (im protocol used by jabber, google talk)
IMHO : should be optional
see in the maven generated site :
http://scala-tools.org/mvnsites/liftweb/lift-core/dependencies.html
there is the list of dependencies (direct, indirect (via a tree)) and a brief description (if lib's developer provide it)
There is some advantages to use maven ;) (like this report)
>
> A couple of quick notes:
>
> * you probably know this, but lift does not run with scala 2.7x RC.
> Apparently the scala library has moved their StringBuilder from compat
> to the scala root.
it is a work, in progress, patches are welcome
Thanks for the feedback.
/davidB
>
> >
David Pollak
The serialization issue is known. I've been trying to track down what in Scala is not serializable in order to make this stuff transient. It's slow going.
In terms of XMPP and AMQP support, they are a core part of lift just as RDBMS support is a core part of lift. Over the next few years, I expect that we'll all see real-time web applications as the norm just as CRUD apps are the norm on the web today. lift is at the front of that wave.
Thanks,
David
Robert Koberg
On Tue, 2008-02-12 at 23:19 +0100, David Bernard wrote:
<snip-agree/>
>
> see in the maven generated site :
> http://scala-tools.org/mvnsites/liftweb/lift-core/dependencies.html
> there is the list of dependencies (direct, indirect (via a tree)) and a brief description (if lib's developer provide it)
> There is some advantages to use maven ;) (like this report)
That is pretty cool. That and the convergence look really helpful. It
seems the links on the dependency convergence don't work?
In compiling and deploying the simple webapp, the only jars that are
necessary are:
commons-fileupload-1.2.jar
hellolift2.jar
lift-core-0.6-SNAPSHOT.jar
log4j-1.2.12.jar
scala-library.jar
It seems that fileupload and log4j should be optional as well. From a
quick look at how fileupload in RequestState, it looks a little
worrying. Is it a potential vector of attack? It appears every request,
after a check for POST plus an XML stream, is checked to see if files
are being passed.
>
> >
> > A couple of quick notes:
> >
> > * you probably know this, but lift does not run with scala 2.7x RC.
> > Apparently the scala library has moved their StringBuilder from compat
> > to the scala root.
>
> it is a work, in progress, patches are welcome
That seems like something I can do :) assuming it was a simple move and
not an api change.
best,
-Rob
David Pollak
Robert Koberg wrote:
On Tue, 2008-02-12 at 23:19 +0100, David Bernard wrote: <snip-agree/>see in the maven generated site : http://scala-tools.org/mvnsites/liftweb/lift-core/dependencies.html there is the list of dependencies (direct, indirect (via a tree)) and a brief description (if lib's developer provide it) There is some advantages to use maven ;) (like this report)That is pretty cool. That and the convergence look really helpful. It seems the links on the dependency convergence don't work? In compiling and deploying the simple webapp, the only jars that are necessary are: commons-fileupload-1.2.jar hellolift2.jar lift-core-0.6-SNAPSHOT.jar log4j-1.2.12.jar scala-library.jar It seems that fileupload and log4j should be optional as well. From a quick look at how fileupload in RequestState, it looks a little worrying. Is it a potential vector of attack?
And fileupload is not optional (and neither are Mail or Activation).
log4j may be excluded if you've plugged in another logging mechanism, but the default lift logging is done with log4j.
Robert Koberg
On Tue, 2008-02-12 at 15:16 -0800, David Pollak wrote:
> >
> > It seems that fileupload and log4j should be optional as well. From a
> > quick look at how fileupload in RequestState, it looks a little
> > worrying. Is it a potential vector of attack?
> Can you describe the attack?
$ curl -F upload=@downloads/scala-downl/SeaJUG.mov
localhost:8080/hellolift2/index.html
<html><head><title>Apache Tomcat/6.0.16 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.servlet.ServletException: Filter execution threw an exception
</pre></p><p><b>root cause</b> <pre>java.lang.OutOfMemoryError: Java
heap space
java.util.Arrays.copyOf(Arrays.java:2786)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
net.liftweb.util.Helpers$.readOnce$1(Helpers.scala:970)
net.liftweb.util.Helpers$.readWholeStream(Helpers.scala:974)
net.liftweb.http.RequestState$$anon
$1.next(RequestState.scala:67)
net.liftweb.http.RequestState$$anon
$1.next(RequestState.scala:62)
scala.Iterator$class.toList(Iterator.scala:611)
net.liftweb.http.RequestState$$anon
$1.toList(RequestState.scala:62)
net.liftweb.http.RequestState$.apply(RequestState.scala:69)
net.liftweb.http.LiftFilter.doFilter(LiftServlet.scala:739)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is
available in the Apache Tomcat/6.0.16 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache Tomcat/6.0.16</h3></body></html>
David Pollak
Anyone else have thoughts on this?
--
lift, the secure, simple, powerful web framework http://liftweb.net
Collaborative Task Management http://much4.us
Steve Jenson
blocked at the border.
Marius
On Feb 13, 7:59 am, "Steve Jenson" <ste...@gmail.com> wrote:
> The major difference is that a standard DoS can be detected and
> blocked at the border.
the thread pools. Container thread pools can not prevent the scenario
mentioned by David P. where too many HTTP sessions accumulates =>
OOME.One way that could prevent OutOfMemoryError's would by that
LiftSession to use SoftReference's but then again when gc purges the
SoftReference's the LiftSession itself is in a limbo state.
Now specifically regarding the file upload scenarios lift could
probably use the file-system temporary and not keep the uploaded file
in memory (Sort of NIO Memory-Mapped file mechanism or even something
simpler)
Viktor Klang
-V
_____________________________________
/ \
/lift/ committer (www.liftweb.net)
SGS member (Scala Group Sweden)
SEJUG member (Swedish Java User Group)
\_____________________________________/
Robert Koberg
On Wed, 2008-02-13 at 01:01 -0800, Marius wrote:
> Now specifically regarding the file upload scenarios lift could
> probably use the file-system temporary and not keep the uploaded file
> in memory (Sort of NIO Memory-Mapped file mechanism or even something
> simpler)
I am not really understanding why fileupload is so tightly integrated
for *each* request. If I needed it (mostly I never would), I would want
to have it for a specific request URI. I really think this should be
optional. What is the rational for having it so tightly integrated?
If anything should be woven that tightly, a better option would be PUT
(and DELETE). Hopefully that would not mean bringing in all of Slide,
but most containers seem to generally handle it well and there is always
httpd.
best,
-Rob
Marius
On Feb 13, 2:06 pm, Robert Koberg <r...@koberg.com> wrote:
> On Wed, 2008-02-13 at 01:01 -0800, Marius wrote:
> > Now specifically regarding the file upload scenarios lift could
> > probably use the file-system temporary and not keep the uploaded file
> > in memory (Sort of NIO Memory-Mapped file mechanism or even something
> > simpler)
>
> I am not really understanding why fileupload is so tightly integrated
> for *each* request. If I needed it (mostly I never would), I would want
> to have it for a specific request URI. I really think this should be
> optional. What is the rational for having it so tightly integrated?
uses commons-fileupload to handle file upload scenarios but is also
using it (apparently) to verify if there is a multipart content.
commons-fileupload provides convenient ways of doing such things and
it's documented as being a lift dependency .... so why not use it?
Maybe (and this is David's P. decision) mutipart verification
(ServletFileUpload.isMultipartContent) later on IF the current request
is not regular get/post request .I'm refering to RequestState line
61 ... to be moved as the latest verification. In this way for regular
HTTP request commons-fileupload is not referenced. (Of course unless
I'm missing something :) ...)
Josh Berry
enough information. Please let me know if I've done so.
On Feb 13, 2008 10:10 AM, Marius <marius...@gmail.com> wrote:
>
>
>
> On Feb 13, 2:06 pm, Robert Koberg <r...@koberg.com> wrote:
> > On Wed, 2008-02-13 at 01:01 -0800, Marius wrote:
> > > Now specifically regarding the file upload scenarios lift could
> > > probably use the file-system temporary and not keep the uploaded file
> > > in memory (Sort of NIO Memory-Mapped file mechanism or even something
> > > simpler)
> >
> > I am not really understanding why fileupload is so tightly integrated
> > for *each* request. If I needed it (mostly I never would), I would want
> > to have it for a specific request URI. I really think this should be
> > optional. What is the rational for having it so tightly integrated?
>
> Ok ... it doesn't appear to me that lift does wrong things here. It
> uses commons-fileupload to handle file upload scenarios but is also
> using it (apparently) to verify if there is a multipart content.
> commons-fileupload provides convenient ways of doing such things and
> it's documented as being a lift dependency .... so why not use it?
I believe the concern is less over the fact that it uses it to verify
the multipart content, but that it appears to have a parse first and
decide if necessary later approach. That is to say, if a multipart
POST request comes to a URI that is not designed to handle file
uploads, it should not get parsed at all.
David Pollak
On Feb 13, 7:59 am, "Steve Jenson" <ste...@gmail.com> wrote:
> The major difference is that a standard DoS can be detected and
> blocked at the border.
Well DoS is limited by FW and containers add their contribution using
the thread pools. Container thread pools can not prevent the scenario
mentioned by David P. where too many HTTP sessions accumulates =>
OOME.One way that could prevent OutOfMemoryError's would by that
LiftSession to use SoftReference's but then again when gc purges the
SoftReference's the LiftSession itself is in a limbo state.
Now specifically regarding the file upload scenarios lift could
probably use the file-system temporary and not keep the uploaded file
in memory (Sort of NIO Memory-Mapped file mechanism or even something
simpler)
or there can be a per application cap on the size of the file along with lazy resolution of the parameters.
Robert Koberg
On Wed, 2008-02-13 at 07:10 -0800, Marius wrote:
>
>
> On Feb 13, 2:06 pm, Robert Koberg <r...@koberg.com> wrote:
> > On Wed, 2008-02-13 at 01:01 -0800, Marius wrote:
> > > Now specifically regarding the file upload scenarios lift could
> > > probably use the file-system temporary and not keep the uploaded file
> > > in memory (Sort of NIO Memory-Mapped file mechanism or even something
> > > simpler)
> >
> > I am not really understanding why fileupload is so tightly integrated
> > for *each* request. If I needed it (mostly I never would), I would want
> > to have it for a specific request URI. I really think this should be
> > optional. What is the rational for having it so tightly integrated?
>
> Ok ... it doesn't appear to me that lift does wrong things here. It
> uses commons-fileupload to handle file upload scenarios but is also
> using it (apparently) to verify if there is a multipart content.
> commons-fileupload provides convenient ways of doing such things and
> it's documented as being a lift dependency .... so why not use it?
Why not just (well, the lift/scala equivalent):
String contentType = servletRequest.getContentType();
if (contentType.toLowerCase().indexOf("multipart") >= 0) {
...
best,
-Rob
David Pollak
What are the failure cases in this scenario?
If you use the standard lift mechanisms for binding form elements to actions and someone sends you a file rather than a String for one of your form elements, it will just be ignored. If you're using lift to compose your forms, you will never be in a situation where you can bind a file upload field (with requires a different kind of function to be bound to it) to something that's not expecting a file upload.
The worst case is an out of memory error which I'm going to patch in a few minutes.
The second to worst case is loading a bunch of bytes that will be discarded, but there are half a dozen ways to cause a server to load lots of bytes and do early processing of those bytes. You can just POST a huge form to a server and it's going to parse the bytes and probably discard them, so the fact that the bytes being parsed are from a multipart form rather than a url encoded for makes little difference that I can see.
Thanks,
David
--
David Pollak
On Wed, 2008-02-13 at 01:01 -0800, Marius wrote:
> Now specifically regarding the file upload scenarios lift could
> probably use the file-system temporary and not keep the uploaded file
> in memory (Sort of NIO Memory-Mapped file mechanism or even something
> simpler)
I am not really understanding why fileupload is so tightly integrated
for *each* request. If I needed it (mostly I never would), I would want
to have it for a specific request URI. I really think this should be
optional. What is the rational for having it so tightly integrated?
Because I see absolutely no difference between submitting a URL Encoded form and a multipart form. The fact that most framework treat them differently is a defect of those frameworks. Information submitted to a web application should be normalized before it gets to the application rather than being treated differently.
If anything should be woven that tightly, a better option would be PUT
(and DELETE). Hopefully that would not mean bringing in all of Slide,
but most containers seem to generally handle it well and there is always
httpd.
lift has excellent support for REST. Implementing PUT/DELETE, etc. is super simple:
case RequestMatcher(r, ParsePath("webservices" :: id :: _, _,_), DeleteRequest, _) => doDeleteForId(id)
Thanks,
David
best,
-Rob
--
Robert Koberg
On Wed, 2008-02-13 at 09:17 -0800, David Pollak wrote:
>
> Now specifically regarding the file upload scenarios lift
> could
> probably use the file-system temporary and not keep the
> uploaded file
> in memory (Sort of NIO Memory-Mapped file mechanism or even
> something
> simpler)
>
> or there can be a per application cap on the size of the file along
> with lazy resolution of the parameters.
OK, forgive me if I am being dense, but why load it at all if you are
not explicitly expecting it? Does this have something to do with xmpp or
something else that I am missing?
best,
-Rob
David Pollak
On Wed, 2008-02-13 at 07:10 -0800, Marius wrote:
>
>
> On Feb 13, 2:06 pm, Robert Koberg <r...@koberg.com> wrote:
> > On Wed, 2008-02-13 at 01:01 -0800, Marius wrote:
> > > Now specifically regarding the file upload scenarios lift could
> > > probably use the file-system temporary and not keep the uploaded file
> > > in memory (Sort of NIO Memory-Mapped file mechanism or even something
> > > simpler)
> >
> > I am not really understanding why fileupload is so tightly integrated
> > for *each* request. If I needed it (mostly I never would), I would want
> > to have it for a specific request URI. I really think this should be
> > optional. What is the rational for having it so tightly integrated?
>
> Ok ... it doesn't appear to me that lift does wrong things here. It
> uses commons-fileupload to handle file upload scenarios but is also
> using it (apparently) to verify if there is a multipart content.
> commons-fileupload provides convenient ways of doing such things and
> it's documented as being a lift dependency .... so why not use it?
Why not just (well, the lift/scala equivalent):
String contentType = servletRequest.getContentType();
if (contentType.toLowerCase().indexOf("multipart") >= 0) {
Because the multipart stuff also does the parsing of the multipart form and that's the tough thing.
David Pollak
It has to do with the fact that request parameters are uniformly available to the lift application. What is the problem with this?
best, -Rob