State and binding

[Alex]

If I have this in a (not Stateful) Snippet, is the checkAndSave
closure saving state on the server? Where is the reference to foo
saved?


def add(form: NodeSeq) : NodeSeq = {
val foo = Foo.create
def checkAndSave(): Unit = foo.validate match {
case Nil => foo.save ; S.notice("Started foo: " + foo.body)
case xs => S.error(xs) ; S.mapSnippet("Foo.add", doBind)
}
def doBind(form: NodeSeq) =
bind("foo", form,
"body" -> foo.body.toForm,
"submit" -> submit("Submit", checkAndSave))

doBind(form)
}

[marius d.]

The foo reference is captured in checkAndSave which is saved on
LiftSession in function mapping.

Br's,
Marius

[Alex]

Perhaps this debate has been had before but I didn't see it. I have a
feeling this might be contentious.

Doesn't this violate the basic tenet that you should not use server
side state when you don't have to? There is no reason for instance
when someone is submitting a news group posting to bind an empty
posting in a closure and store it on the server, thereby breaking the
application in any number of cases:

1) The session times out because the user got distracted or lost their
connection temporarily
2) The user abuses the back button
3) The server fails over (assuming there is no memory replication)
4) There is a temporary connection failure and the user reloads
5) The server is restarted

All of the above cases are recoverable with what, IMO, are properly
designed frameworks that minimize server-side state. Those are the
things that taught us state should be stored in the client whenever
possible.

I saw a bit of a discussion distinguishing REST from human
interaction, saying human interactions are not stateless, but that
does not excuse the application being so vulnerable to breakage.

This reminds me of why I dislike many ASP applications I've used -
they always break when users don't behave well or when some kicks a
server - and users, myself included, always misbehave. I use the back
button when I shouldn't, I leave forms half filled out when I answer
the phone, and as a frequent traveler I tend to use web applications
from places with shoddy internet. I also don't like to lose my work
when a server fails over or is restarted.

There may be a few gains from designing things the lift way -
simplicity, security, etc - but they seem to come at too great a cost.

There maybe some solutions to this, such as serialization of this data
to the client, but I'm not sure I like any of them.

-Alex

[David Pollak]

Alex,

If you don't like this design, then you will not like Lift.

I have been doing web development since 1995 and building web frameworks since 1996.  I have seen a lot of different approaches to web frameworks and the two web frameworks that have the best approach are WebObjects and Seaside.  The worst approaches are Rails and the "share nothing" ilk.  Lift borrows from Seaside and WebObject.  Lift seeks to abstract away the HTTP request/response cycle.  State is maintained on the app server, not pawned off to some other place.  This leads to better scalability characteristics.  The developer deals with higher level abstractions than dissecting the HTTP request and reconstituting state.

Benefits:

• Higher levels of developer productivity
• Fewer defects
• Much more secure sites (try doing a replay attack or parameter tampering on a Lift app)
• More scalable sites (there's no single point of "state")

So, in my experience and to my mind, the "tenet" of minimizing server-side state is broken and wrong.  State should placed where it is (1) most secure and (2) leads to the most responsive apps.

Thanks,

David

PS -- The reason that everybody hits a wall with moderate volume Rails apps is that the fantasy of stateless stacks devolve into a pile of dog poop when actually tested against the amazingly fragile run-time that is Ruby.

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

[marius d.]

On Mar 24, 6:53 pm, Alex <a...@liivid.com> wrote:
> Perhaps this debate has been had before but I didn't see it.  I have a
> feeling this might be contentious.
>
> Doesn't this violate the basic tenet that you should not use server
> side state when you don't have to?

No. As long as you are binding a function we are talking about state.

>  There is no reason for instance
> when someone is submitting a news group posting to bind an empty
> posting in a closure and store it on the server, thereby breaking the
> application in any number of cases:
>
> 1) The session times out because the user got distracted or lost their
> connection temporarily
> 2) The user abuses the back button
> 3) The server fails over (assuming there is no memory replication)
> 4) There is a temporary connection failure and the user reloads
> 5) The server is restarted

I don't see how that would break the application. The function mapping
is cleared is the session times out or it is terminated. Furthermore
Lift has a garbage collection mechanism that removes the functions
that are not utilized.
None of the above cases breaks a Lift application ... if the
application itself is correctly designed and implemented. If we are
talking about clustered applications Lift currently uses sticky
sessions mechanism meaning that all requests pertaining to a given
session are processed by the same node. This can be easily ensured by
load balancing rules. If a node breaks part of the conversation state
can be ensured by application design when processed by a different
node ...of course in the context of a different HttpSession.

>
> All of the above cases are recoverable with what, IMO, are properly
> designed frameworks that minimize server-side state.  Those are the
> things that taught us state should be stored in the client whenever
> possible.

Yes and no. I agree with state minimization but adding burden to
client side doesn't solve much in many cases. With Lift and Scala we
are leveraging functions and partial functions to process requests. If
you want that bad purely stateless handling you can also do that with
Lift. Please see LiftRules.statelessDispatchTable.

>
> I saw a bit of a discussion distinguishing REST from human
> interaction, saying human interactions are not stateless, but that
> does not excuse the application being so vulnerable to breakage.

I still don't get where your vulnerability comes from.

>
> This reminds me of why I dislike many ASP applications I've used -
> they always break when users don't behave well or when some kicks a
> server - and users, myself included, always misbehave.  I use the back
> button when I shouldn't, I leave forms half filled out when I answer
> the phone, and as a frequent traveler I tend to use web applications
> from places with shoddy internet.  I also don't like to lose my work
> when a server fails over or is restarted.

With a bit of care for design this can be taken care. For instance if
you have an Ajax form and the request fails, Lift will try to resend
that ... if the retry request qoes to a new node then we are talking
about the application design if the application has enough contextual
information to handle that or not. Such as in forms that do not
require logged in users (i.e. posting a comment on a blog or forum as
anonymous) the application can safely process the form request in the
context of a new HttpSession/LiftSession.

>
> There may be a few gains from designing things the lift way -
> simplicity, security, etc - but they seem to come at too great a cost.

What is this "great cost" ... one can write poor designed applications
in any language or framework on this Earth (dunno about other
planets). A framework can only do so much but it does not guarantee
flawless applications.

>
> There maybe some solutions to this, such as serialization of this data
> to the client, but I'm not sure I like any of them.

That would be a performance killer.

[Charles F. Munat]

I remain confused by a lot of this stuff.

Here is an example:

I have a site that creates pages dynamically based on information in a
database. A good example is bio pages. Users add their bios, and then
the bios are available through a URL.

Now if I use, for example, the user's ID in the URL, thus:

members/666

Then it is a simple matter of a lookup in the database and spit out the
code. The page is easily bookmarked and returned to. It's even
reasonably human readable (members/beelzebub would be even better). I
can do some simple things to prevent scripting attacks. I've used this
system for many years and have never had a problem.

If I use a closure instead I end up with something like this:

members/?F92019795619530=_

Can't be bookmarked. Not easily human readable. Expires. For me, this is
essentially unworkable.

I see the benefit to continuations, but they don't seem like the best
solution in every instance. Even in a simple CRUD app, I like being able
to bookmark the edit page for an item and come back to it. Where I tend
to use continuations is for deletes, or for pages I specifically want to
expire.

So what is the best practice here, and why? How can I create my dynamic
member pages and make them bookmarkable and non-expiring? Is there a
Lift way that I'm missing?

Chas.

[David Pollak]

No.  You got the balance just right.  For stuff that you bookmark and things that are stable, use stable identifiers.  That's totally cool.  But when you're building a page with a form, you should worry about the name of the fields on the form.  You should only have to worry about what happens when the form is submitted. But the user never bookmarks the form field.  The user never looks at the form (unless they're a hyper-geek).

Bright line: for stuff that's intended for bookmarking or for stuff that conveys meaning to a human, generate the stable identifier yourself and deal with the HTTP request appropriately (e.g., a rewrite rule.)  For the plumbing and stuff that's related to sessions, let Lift generate GUIDs and associate them with functions.

Cool?

[Charles F. Munat]

Glad to see I have been using best practices... I am in complete
agreement about form field names, and I like the added benefit that
they're obscured.

One area where the continuations are very useful is in multipage forms.
I like the idea of being able to, for example, stop halfway through a
multipage survey and come back to it later. But for this, the
continuation has to be a little more permanent. Is there any facility
built into Lift for this?

I played around with Seaside briefly until I couldn't stand the Squeak
IDE anymore (one seriously ugly application -- looks like it was
designed by Mickey Mouse). But I thought that the continuations were
persistent. Years and years ago I used to work with Apache Cocoon, which
was my first exposure to continuations (I'm such a newbie), and again,
you could stop and restart a process at any time.

Chas.

> <mailto:marius.dan...@gmail.com>> wrote:
> >>
> >>> The foo reference is captured in checkAndSave which is saved on
> >>> LiftSession in function mapping.
> >>> Br's,
> >>> Marius
> >>> On Mar 24, 10:23 am, Alex <a...@liivid.com

[Alex]

> >  There is no reason for instance
> > when someone is submitting a news group posting to bind an empty
> > posting in a closure and store it on the server, thereby breaking the
> > application in any number of cases:
>
> > 1) The session times out because the user got distracted or lost their
> > connection temporarily
> > 2) The user abuses the back button
> > 3) The server fails over (assuming there is no memory replication)
> > 4) There is a temporary connection failure and the user reloads
> > 5) The server is restarted
>
> I don't see how that would break the application. The function mapping
> is cleared is the session times out or it is terminated. Furthermore
> Lift has a garbage collection mechanism that removes the functions
> that are not utilized.
> None of the above cases breaks a Lift application ... if the
> application itself is correctly designed and implemented.

How does this NOT break an application? If the session is expired and
the function mapping is cleared, what happens when the form is
submitted? Breakage.... If someone is in the middle of writing a
long essay, or building a travel reservation (this happens to me all
the f'in time on most travel sites) then it's gonzo. "Sorry we lost
all your work. Please start over." The alternative is to persist in
the database which is worse than on the client.

If you can tell me how an application can be "correctly designed and
implemented" to avoid this I'd be very interested. I am not
diametrically opposed to this, but I don't see how it's avoidable in
any reasonable way.

>
> > All of the above cases are recoverable with what, IMO, are properly
> > designed frameworks that minimize server-side state.  Those are the
> > things that taught us state should be stored in the client whenever
> > possible.
>
> Yes and no. I agree with state minimization but adding burden to
> client side doesn't solve much in many cases.

Yes, it does.

With 'stateless' frameworks:

1) Form submits: post = "Long agonizing post"
2) Session is expired so user is redirected to login page. Post is
stored temporarily.
3) User logs in and is redirected to posting page filled out with
their post.
4) User resubmits

With Lift

1) Form submits: B2345235DSFGA = "Long agonizing post"
2) Function mapping has been cleared. Application panics and has no
idea what that junk was.

> With a bit of care for design this can be taken care. For instance if
> you have an Ajax form and the request fails,

And if the session is expired, the entire client page is invalid and
the page has to be reloaded...

[Charles F. Munat]

I think that the two of you are using "break" in different ways. Alex,
you mean it doesn't do what the user expects it to do. Marius, I think,
means that the application crashes.

I agree that this is a nasty behavior. One could say that it's the
user's responsibility to keep track of their session time, but I don't
subscribe to that. I try to make my sites as user friendly as possible.

Another scenario is that the session doesn't expire, and the user walks
away from the terminal without closing the browser. Then I sit down,
find his half finished form and finish it for him. Isn't that equally
broken? Maybe more so?

The issue here, as I see it, has little to do with continuations per se,
and much to do with application design. On my sites, for example, when a
user requests a page, the uri of the page is stored. If the user is not
authenticated -- possibly because the session timed out -- then she is
redirected to the login page. But after login, the site redirects to the
original page that she requested.

There is no reason you can't temporarily store form data -- if I can
store a GET, you can store a POST -- and then if the user chooses to log
back in, redirect and repopulate the form, or even provide a
confirmation dialog asking if he/she wants to go ahead and post the form
data. Why more sites don't do this is beyond me. It's not a difficult
programming task.

I'd love to see something like that built in to Lift (or at least made
very easy).

Another option is to store form data as it's entered. I have a survey
site that does this right now (albeit with some bugginess). Put it in a
temporary table. Then when the user logs back in, give him the option to
return to the form and complete it, or delete the partial form data, or
leave it for later.

There are lots of ways to make sites better without giving up the power
of continuations.

Chas.

[Alex]

> State is maintained on the app server, not pawned
> off to some other place.  This leads to better scalability characteristics.
>  The developer deals with higher level abstractions than dissecting the HTTP
> request and reconstituting state.

I have just seen to many "Sorry your session expired and we lost
everything" messages in my lifetime. Those result in lost sales, lost
leads, lost posts, lost users, and just general pain, and they are
usually unnecessary since there is often very little state.

I would love to be able to work the way Lift is designed - it's
definitely better for the developer. It's that I want to be forgiving
to the way the web works. With lift the state doesn't seem
recoverable, even in cases where it is trivial to do so, or where the
entire interesting state was an empty object.

> PS -- The reason that everybody hits a wall with moderate volume Rails apps
> is that the fantasy of stateless stacks devolve into a pile of dog poop when
> actually tested against the amazingly fragile run-time that is Ruby.

I came to Scala because I didn't like Rails or Ruby, so no argument
about that part. I did like Merb though.

Most apps I have written were stateless inasmuch as they could be. I
agree it's not always fun, but it makes things far more reliable. I
can go back to a form generated 2 weeks ago and submit it and it still
works. You don't always want that but usually you do.

For instance, if I'm on the home page of a reservation system and
haven't reloaded for 2 weeks, I put in my dates of travel and cities,
I want it to work. I don't want an error page and then have it be
incapable of even redisplaying the form with my fields filled out.
You will lose a large percentage of users at that point.
Unfortunately that's the way most travel sites work.

Believe me, I would love the above to not be the case or if there is
some workaround you can suggest. I love Scala and I like a lot of
things about Lift so far.

[Alex]

>
> I agree that this is a nasty behavior. One could say that it's the
> user's responsibility to keep track of their session time, but I don't
> subscribe to that. I try to make my sites as user friendly as possible.

Try telling anyone in marketing, user experience, or anyone not in
tech that it's user's fault they didn't keep track of their
session..... ick.

If 100ms load time can affect stickiness (as studies have revealed)
imagine what returning an error or no results from a search just
because the user didn't search in time.

I for one know I dumped several travel sites because of their poor
behavior in this regard.

>
> Another scenario is that the session doesn't expire, and the user walks
> away from the terminal without closing the browser.

You still expire the session and force the user to login. You just
avoid losing their work.

> There is no reason you can't temporarily store form data -- if I can
> store a GET, you can store a POST -- and then if the user chooses to log

> back in, redirect and repopulate the form ... Why more sites don't do this is beyond me. It's not a difficult
> programming task.

As far as I can tell, this is not possible with lift. The association
of the POST data with any particular object or field is gone once the
session expires so the submitted values are useless.

>
> Another option is to store form data as it's entered. I have a survey
> site that does this right now (albeit with some bugginess). Put it in a
> temporary table. Then when the user logs back in, give him the option to
> return to the form and complete it, or delete the partial form data, or
> leave it for later.

Auto-save or manually saving to a database is fine, but I don't think
it address the issue.

[David Pollak]

On Tue, Mar 24, 2009 at 5:38 PM, Alex <al...@liivid.com> wrote:

With Lift

1) Form submits: B2345235DSFGA = "Long agonizing post"
2) Function mapping has been cleared.  Application panics and has no
idea what that junk was.

1. This is an empirically wrong statement.  Test it out.
2. A fair number of Lift apps are in the wild and the kind of problem that you're imaging is very rare.
3. When this kind of problem does happen (the front page of the travel site which is a stateless page rather than something that is part of a user's session), there are simple ways to deal with the situation, but these are exceptions rather than the rule.
4. There is nothing in Lift that stops you from putting stable form field names on a page and extracting them with S.param(field_name)
5. This is Lift's design... to make the common things easier and to allow the developer to fall back to guts of HTTP when necessary.
6. This list is for helping newbies become seasoned and for learning from people's real-world use to Lift in order to enhance Lift.
7. I'd suggest not pushing on this issue any more until you've built an app and seen how Lift works in practice.

Thanks,

David

[Chad Skinner]

So, in my experience and to my mind, the "tenet" of minimizing server-side state is broken and wrong.  State should placed where it is (1) most secure and (2) leads to the most responsive apps.

Not know much about lift yet and wanting to learn more, what is stored in the server session for a simple application? I am assuming it is used by the binder to store the generated form field names so the submitted fields can be rebound ... what other state does the framework store in it?

[David Pollak]

On Wed, Mar 25, 2009 at 6:15 AM, Chad Skinner <chadws...@gmail.com> wrote:

So, in my experience and to my mind, the "tenet" of minimizing server-side state is broken and wrong.  State should placed where it is (1) most secure and (2) leads to the most responsive apps.

Not know much about lift yet and wanting to learn more, what is stored in the server session for a simple application? I am assuming it is used by the binder to store the generated form field names so the submitted fields can be rebound ... what other state does the framework store in it?

Any SessionVars are held in Session state.  Bindings from HTML elements to functions are held in Session state.  Bindings between Comet Actors and the HTML the represents them in held in Session state.

 

[Chad Skinner]

Not know much about lift yet and wanting to learn more, what is stored in the server session for a simple application? I am assuming it is used by the binder to store the generated form field names so the submitted fields can be rebound ... what other state does the framework store in it?

Any SessionVars are held in Session state.  Bindings from HTML elements to functions are held in Session state.  Bindings between Comet Actors and the HTML the represents them in held in Session state.

 

Thanks for the information, I believe that the documentation states that lift has its own session system and that it does not use the servers session ... if this is true and you are running two redundant servers, will the servers session replication / clustering copy the Lift session?

State/session replication in a large cluster can cause problems, but for our situation I don't see server state as being a problem as long as it is minimal and replicates in clusters.

I went to the Colorado Software Summit last year and went to a couple of presentations by Yan Pujante and was very impressed by what they are doing at linked-in. He presented a couple of sessions one on OSGi and the issues they are solving or hoping to solve with it as well as the obstacles they have encountered. Also, he did a presentation on their new security / authentication system which was very interesting. One thing he mentioned was that they were moving to a stateless environment, where the only state that is maintained is that of the authenticated user object.

Working for a school district I can say that Linked-in's world is ... well different ... they have over 400 developers we have 2 ... they have I believe he said 600 web servers ... we have two. Server state for us ... not really a problem and I'd be happy if my development is easier, quicker and I can better meet the needs of our users in a timely fashion.

[Viktor Klang]

Hi Chad!

Lift is intended to be clustered using a load-balancer with session-affinity, which means that no session replication is needed unless a node goes down.

From only having almost a decade of web-framework development experience, I fully support the notion of having the session state serverside for highly interactive rich internet applications.
Not only does it simplify development and enhance security, but it enables a whole lot of shortcuts not available for share-nothing approaches.

That being said, I am a very big proponent for the REST model, which Lift is _very_ competent in providing an API for you to use, for ROA/REST needs.

From what you may gather from this e-mail, I strongly believe in using the right tools for each job.

Does this answer help you?

Cheers,
Viktor

--
Viktor Klang
Senior Systems Analyst

[Chad Skinner]

So, are the sessions replicated or do you lose active connections if a node goes down?

I agree I don't have a problem with server session state ... the only problem I have is we originally used JSF for a number of our applications and I hate the reliance on POST in order to submit the serialized state. There is no reason in my mind not to support GET and keep the session information on the server .. For this LIFT seems a good fit.

Not lift related, but does anyone know of an inexpensive load balancer that works well and would support session affinity? The only reason I ask is that I would like to replace our current software LB with a hardware device and with the budget cuts due to the economy I need something that will work well with relatively small loads and that is cost effective.

-- Chad

[David Pollak]

On Wed, Mar 25, 2009 at 7:38 AM, Chad Skinner <chadws...@gmail.com> wrote:

So, are the sessions replicated or do you lose active connections if a node goes down?

Yes.  And your app goes down if the RDBMS goes down.
 

I agree I don't have a problem with server session state ... the only problem I have is we originally used JSF for a number of our applications and I hate the reliance on POST in order to submit the serialized state. There is no reason in my mind not to support GET and keep the session information on the server .. For this LIFT seems a good fit.

Not lift related, but does anyone know of an inexpensive load balancer that works well and would support session affinity? The only reason I ask is that I would like to replace our current software LB with a hardware device and with the budget cuts due to the economy I need something that will work well with relatively small loads and that is cost effective.

I would use Nginx software as a load balancer.  It's fast enough to handle a ton of traffic (on the order of 10K requests per second on a dual opteron).
 

-- Chad

On Wed, Mar 25, 2009 at 8:58 AM, Viktor Klang <viktor...@gmail.com> wrote:

Hi Chad!

Lift is intended to be clustered using a load-balancer with session-affinity, which means that no session replication is needed unless a node goes down.

From only having almost a decade of web-framework development experience, I fully support the notion of having the session state serverside for highly interactive rich internet applications.
Not only does it simplify development and enhance security, but it enables a whole lot of shortcuts not available for share-nothing approaches.

That being said, I am a very big proponent for the REST model, which Lift is _very_ competent in providing an API for you to use, for ROA/REST needs.

From what you may gather from this e-mail, I strongly believe in using the right tools for each job.

Does this answer help you?

Cheers,
Viktor

On Wed, Mar 25, 2009 at 2:46 PM, Chad Skinner <chadws...@gmail.com> wrote:

Not know much about lift yet and wanting to learn more, what is stored in the server session for a simple application? I am assuming it is used by the binder to store the generated form field names so the submitted fields can be rebound ... what other state does the framework store in it?

Any SessionVars are held in Session state.  Bindings from HTML elements to functions are held in Session state.  Bindings between Comet Actors and the HTML the represents them in held in Session state.

 

Thanks for the information, I believe that the documentation states that lift has its own session system and that it does not use the servers session ... if this is true and you are running two redundant servers, will the servers session replication / clustering copy the Lift session?

State/session replication in a large cluster can cause problems, but for our situation I don't see server state as being a problem as long as it is minimal and replicates in clusters.

I went to the Colorado Software Summit last year and went to a couple of presentations by Yan Pujante and was very impressed by what they are doing at linked-in. He presented a couple of sessions one on OSGi and the issues they are solving or hoping to solve with it as well as the obstacles they have encountered. Also, he did a presentation on their new security / authentication system which was very interesting. One thing he mentioned was that they were moving to a stateless environment, where the only state that is maintained is that of the authenticated user object.

Working for a school district I can say that Linked-in's world is ... well different ... they have over 400 developers we have 2 ... they have I believe he said 600 web servers ... we have two. Server state for us ... not really a problem and I'd be happy if my development is easier, quicker and I can better meet the needs of our users in a timely fashion.

--
Viktor Klang
Senior Systems Analyst

[marius d.]

On Mar 25, 3:46 pm, Chad Skinner <chadwskin...@gmail.com> wrote:
> >> Not know much about lift yet and wanting to learn more, what is stored in
> >> the server session for a simple application? I am assuming it is used by the
> >> binder to store the generated form field names so the submitted fields can
> >> be rebound ... what other state does the framework store in it?
>
> > Any SessionVars are held in Session state.  Bindings from HTML elements to
> > functions are held in Session state.  Bindings between Comet Actors and the
> > HTML the represents them in held in Session state.
>
> Thanks for the information, I believe that the documentation states that
> lift has its own session system and that it does not use the servers session
> ... if this is true and you are running two redundant servers, will the
> servers session replication / clustering copy the Lift session?

LiftSession is bound to HTTP session via a Session bridge which
essentially is a HttpSessionBindingListener and
HttpSessionActivationListener. But besides that LiftSession is managed
by a SessionMaster actor.LiftSessions are not replicated among cluster
nodes.