Suggestion: Session API

[mrshoe]

Maintaining session state is extremely common in web applications. In
general, sessions need to be loaded at the beginning of each request,
locked for the duration of the request, and saved at the end of the
request. These operations need to be fast. Ideally you can store any
type of data on the session. The session automatically dies after some
specified expiration time.

People with small sites that run on one box often just store sessions
in memory. To adhere to "shared nothing" and scale to multiple
machines, sometimes the session data is stored in a relational
database. We have developed a dedicated session daemon that meets
these needs quite well (http://pypi.python.org/pypi/pear/0.8).

I don't think that the Data Store is a good place for sessions.
There's no easy way to lock an entity for the duration of a request.
You can't store arbitrary data types (heterogeneous lists,
dictionaries, etc.) in the Data Store, and there is no efficient
serialization module in the sandbox (cPickle, marshal, cjson, etc.),
which would allow fast serialization and storage in a BlobProperty.
There's no clean way to make entities expire and delete themselves
automatically.

I think one of the next tools to be built for App Engine is a session
API that accomplishes what pear does.

shoe

[Brett Morgan]

Session state is one of the main reasons that applications don't scale
out. DataStore is the right place for session state, because by using
DataStore for session state the session information is automatically
replicated across the entire server cluster in a way that obeys
transactional semantics.

The restrictions on GAE are all there to serve the goal of application
scalability.

[mrshoe]

What's needed is something that provides the benefits of DataStore
(some of which you pointed out), but addresses the shortcomings (which
I pointed out).

shoe

[Ian Bicking]

mrshoe wrote:
> I don't think that the Data Store is a good place for sessions.
> There's no easy way to lock an entity for the duration of a request.
> You can't store arbitrary data types (heterogeneous lists,
> dictionaries, etc.) in the Data Store, and there is no efficient
> serialization module in the sandbox (cPickle, marshal, cjson, etc.),
> which would allow fast serialization and storage in a BlobProperty.
> There's no clean way to make entities expire and delete themselves
> automatically.

Are you really storing so much data that pickle will be that slow? Most
of the time sessions contain a handful of strings, and maybe an object
or two, and serializing the object shouldn't be that expensive, even
without cPickle.

Ian

[mrshoe]

Ian,
True enough, and I am indeed using a BlobProperty and pickle at the
moment. I guess the other shortcomings are larger concerns for me.

shoe

[Bryan Donlan]

On Apr 9, 8:19 pm, mrshoe <mrs...@gmail.com> wrote:
> Ian,
> True enough, and I am indeed using a BlobProperty and pickle at the
> moment. I guess the other shortcomings are larger concerns for me.

Why do you need to be doing so much in your session? Why not just
declare the session to be /small/, mostly read-only data, put
encrypted, signed data into a cookie for it, and then if you need
state in some other process pass it along via hidden form elements, or
a reference into a Datastore entity?

[mrshoe]

I'm not doing very much in the session. You don't have to be doing
very much with the session to desire simple features like locking and
automatic expiration. As an author of a web framework that's been used
by myself and others for a couple of years now, I know that these
features are nice to have. It was the one piece of my framework that
was difficult/impossible to port to App Engine.

shoe

[Brett Morgan]

Can you please expand on what you define locking and expiration to be,
so that I can better understand why they can't be implemented on top
of StoreData?

[Brett]

> I don't think that the Data Store is a good place for sessions.
> There's no easy way to lock an entity for the duration of a request.
> You can't store arbitrary data types (heterogeneous lists,
> dictionaries, etc.) in the Data Store, and there is no efficient
> serialization module in the sandbox (cPickle, marshal, cjson, etc.),
> which would allow fast serialization and storage in a BlobProperty.
> There's no clean way to make entities expire and delete themselves
> automatically.

Check out db.Model.get_or_insert()

http://code.google.com/appengine/docs/datastore/modelclass.html#Model_get_or_insert

This will transactionally create or retrieve an entity with a well-
defined key name that you get to choose (in this case, it could be an
ID you associate with the user).

You can then quickly get your session state by the key name and a
Get() call, which is quick.

If you need to write that session entity, you can write a more complex
transaction to do so. However, if the entity is tied to a specific
user, you often won't need to do so, since they will rarely be having
a race condition with themselves.

[mrshoe]

Users having race conditions with themselves is exactly why you want
the session locked for the duration of the request. It does happen.
This is why CherryPy's session tool has hooks for locking.

Re: expiration, sessions are very temporary. You want them to expire
automatically after a specified time, e.g. to force the user to log in
again. That's easy enough to implement using the DataStore. However,
you also don't want them sitting in the DataStore forever. I was
discussing this with my friend at Google who worked on AppEngine, and
his only recommendation for now was to have a cron job on some other
machine that hits a private URL in your app that deletes old sessions.
That's not exactly elegant. A session API that killed sessions after a
specified TTL solves both of those problems.

shoe

On Apr 9, 10:08 pm, Brett <brettalici...@gmail.com> wrote:
> > I don't think that the Data Store is a good place for sessions.
> > There's no easy way to lock an entity for the duration of a request.
> > You can't store arbitrary data types (heterogeneous lists,
> > dictionaries, etc.) in the Data Store, and there is no efficient
> > serialization module in the sandbox (cPickle, marshal, cjson, etc.),
> > which would allow fast serialization and storage in a BlobProperty.
> > There's no clean way to make entities expire and delete themselves
> > automatically.
>
> Check out db.Model.get_or_insert()
>

> http://code.google.com/appengine/docs/datastore/modelclass.html#Model...

[dimrub]

On Apr 10, 3:24 am, Bryan Donlan <bdon...@gmail.com> wrote:
> Why do you need to be doing so much in yoursession? Why not just

> declare thesessionto be /small/, mostly read-only data, put

> encrypted, signed data into a cookie for it, and then if you need
> state in some other process pass it along via hidden form elements, or
> a reference into a Datastore entity?

There are situations in which one is forced to store substantial data
in the session. An example is an application I'm currently working on
(well, more like thinking on, but anyway) in which the user first
uploads a CSV file, then asked to map the columns to some known set of
data fields. I don't want at this point to have the user upload the
file again, so I have to store it somewhere, right? Session store
seems to be the most logical place for that.

[Brett Morgan]

The DataStore is the right place for it. Logic in GAE land is
different to the logic in traditional web app land. It looks the same,
but it ain't.