Memcache sessions - need set time

[Anton Danilchenko]

Hello!

I have found that we set data to memcache on unlimited time:

# line 314

memcache.set(self.sid, pdump, namespace='')  # may fail if memcache is down

We need set this session data to limited time, specified in constructor (for example, on 7 days). After this, we have cleaned old sessions automatically and, if user open page - then data fetched from database and copied to memcache on next 7 days. This is useful improvement for save memcache resources.

Solution is:

memcache.set(self.sid, pdump, namespace='', time=CURRENT_DATETIME + 7DAYS)

And also, I propose add column to SessionModel with automatically updated datetime property. This help us select all expired sessions and delete his. For example, we can do something like this in delete_expired_sessions() method:

SessionModel.filter("date <", CURRENT_DATETIME - 7DAYS).limit(1000).delete()

Please make this two improvements as soon as posible. If you need help - please tell me.

Regards,

Anton.

[Anton Danilchenko]

I found that you have delete this sessions from memcache and datastore by selecting records keys. But I think that this is slow place for the hight-load projects.

[David Underhill]

Are you talking about the code in the __clear_data(self) method?  If so, this method is only called when the session is terminated, or if the session ID is regenerated (using if you need to transition a session from an HTTP to HTTPS connection securely).  These events are uncommon, and so I this won't have a significant impact on performance.

delete_expired_sessions() is the only other place sessions are deleted form datastore and memcache.  This doesn't impact user experience (due to performance) at all - it only needs to be periodically run by a cronjob (which shouldn't disturb users).

~ David

[David Underhill]

On Sat, Dec 11, 2010 at 15:39, Anton Danilchenko <anton.da...@gmail.com> wrote:

Hello!

I have found that we set data to memcache on unlimited time:

# line 314

memcache.set(self.sid, pdump, namespace='')  # may fail if memcache is down

We need set this session data to limited time, specified in constructor (for example, on 7 days). After this, we have cleaned old sessions automatically and, if user open page - then data fetched from database and copied to memcache on next 7 days. This is useful improvement for save memcache resources.

Solution is:

memcache.set(self.sid, pdump, namespace='', time=CURRENT_DATETIME + 7DAYS)

Thanks Anton, this is a great catch.  I'll patch the source code momentarily (the 'time' parameter should be set to get_expiration()).

 

And also, I propose add column to SessionModel with automatically updated datetime property. This help us select all expired sessions and delete his. For example, we can do something like this in delete_expired_sessions() method:

SessionModel.filter("date <", CURRENT_DATETIME - 7DAYS).limit(1000).delete()

I thought about this approach too, but I decided on a more efficient approach.  Instead of adding a date field (which would require its own index and thus slow down datastore writes a bit), the expiration time is stored as part of the entity's key.  This allows us to efficiently find expired sessions without a date field (and its overheads).

The delete_expired_sessions() method demonstrates how this approach works.

~ David

[David Underhill]

I've released gae-sessions v1.06 which fixes the issue Anton discovered - memcache entries are now set to automatically expire along with their session (rather than languishing until memcache evicts them itself).

v1.06 release notes and downloads:

https://github.com/dound/gae-sessions/wiki

~ David

[Anton Danilchenko]

Hello Dvid!

Thank you for super-fast feedback and release! I have integrated your gaesession to my project http://gaeframework.com - you can try to use this feature in the SVN trunk. Now we need implement manual user registration instead of use Google Account. All users can register on our site and login. You sessions help us in this.

About datetime column. I see on this situation with the next vision:

1) I have logged in to site, and my session expires, for example, after 1 hour

2) after 5 minutes I have refresh website page. Now my session should be automatically updated to next hour (expired after one hour, not 55 minutes!)

As I think, now we have set session only on the one hour. And if I have open site in this hour - my session not updated (because KEYs have not updated in the datastore - we can only delete old record and create new record with the new KEY). If this is true - please, use datetime column for SessionModel. This is a bit more data stored in the database, but this add for us more flexibility - we can easily delete expired sessions. In result - we have more saved server resources. And also - disk space for the datastore is very low-cost, As I discussed with Gooogle App Engine main developer - we need to use more disc space and save processor time - this is save our money and allow us to use more requests per second.

Please feedback to me about this proposition. I am really want to improve your library, and I want to learn more about current solutions - maybe I have see not all details of your project.

Thanks!

[Anton Danilchenko]

I propose to do automatically setting up of Cookie key. See how I does it in this code http://code.google.com/p/appengine-framework/source/browse/trunk/src/appengine_config.py?r=311#4

QUESTION: if I have set Model object to session, what kind of information to be stored? For example, I have next code:

# usaerl logged in by email

user = models.User.get(email="te...@example.com")

# store user object to session

session["user"] = user

In this case, what to be stored in memcache and datastore? Only Model ID for wake-up of object on the next request OR object as it is? I need always get fresh information about user (username, email, etc). If I have store object as it is - this is not a good. I need store only Model ID and wakeup Model on each user request to site.

Please, describe me process of storing information on datastore and memcache with sessions, please. Thanks!

[David Underhill]

On Sun, Dec 12, 2010 at 06:57, Anton Danilchenko <anton.da...@gmail.com> wrote:

I propose to do automatically setting up of Cookie key. See how I does it in this code http://code.google.com/p/appengine-framework/source/browse/trunk/src/appengine_config.py?r=311#4

QUESTION: if I have set Model object to session, what kind of information to be stored? For example, I have next code:

# usaerl logged in by email

user = models.User.get(email="te...@example.com")

# store user object to session

session["user"] = user

In this case, what to be stored in memcache and datastore? Only Model ID for wake-up of object on the next request OR object as it is?

Entities are converted to their protobuf representation.  This is more efficient than letting memcache pickle the object as is.

 

I need always get fresh information about user (username, email, etc). If I have store object as it is - this is not a good.

It seems like username (and perhaps other User information) are good candidates for caching - it is unlikely they will  change often, right?

 

I need store only Model ID and wakeup Model on each user request to site.

If you really want to do this, then just store the some_entity.key().id_or_name().  You can reconstruct the key from that and then retrieve the latest information from the datastore.  Datastore roundtrips are expensive; avoid them unless they are really necessary.

 

~ David

[David Underhill]

On Sun, Dec 12, 2010 at 03:55, Anton Danilchenko <anton.da...@gmail.com> wrote:

About datetime column. I see on this situation with the next vision:

1) I have logged in to site, and my session expires, for example, after 1 hour

2) after 5 minutes I have refresh website page. Now my session should be automatically updated to next hour (expired after one hour, not 55 minutes!)

Automatically renewing sessions like this is a bit of security issue.  If an unauthorized user gains access to your session, then if you automatically renew the session then that unauthorized user will never lose the session.  On the other hand, if you let the session expire instead of renewing it then at least the unauthorized user will lose access eventually.

 

As I think, now we have set session only on the one hour.

Based on the example you provided above, that is correct.

 

And if I have open site in this hour - my session not updated (because KEYs have not updated in the datastore - we can only delete old record and create new record with the new KEY). If this is true - please, use datetime column for SessionModel. This is a bit more data stored in the database, but this add for us more flexibility - we can easily delete expired sessions. In result - we have more saved server resources. And also - disk space for the datastore is very low-cost, As I discussed with Gooogle App Engine main developer - we need to use more disc space and save processor time - this is save our money and allow us to use more requests per second.

Expired sessions are already easy to delete efficiently.  Just use gaesessions.delete_expired_sessions() (ideally in a cronjob which runs periodically).   The exact frequency it should run at depends on the volume of traffic the site receives, but once a day should be a reasonable default for most apps.

 

~ David

[Anton Danilchenko]

I agree that user change information not every day. And we can store object in session. But we need reload this model every time (for example, one time per hour). I can do this manually by setting new information to session. But, I believe that we can simplify this process.

# user not logged in (in main constructr)

if "user" not in self.session:

  self.session["user"] = get_current_user()

# and use this user object in all applications

In this case I have store user in the session on 1 week. If user have changed some information in profile - I have see old information about user? Or, I have see changed information?

# in my application by url /user/change_avatar

user = self.session["user"]

user.avatar = "other_avatar_image.png"

user.put()

In session this information automatically updated?

[David Underhill]

On Sun, Dec 12, 2010 at 07:34, Anton Danilchenko <anton.da...@gmail.com> wrote:

If user have changed some information in profile - I have see old information about user? Or, I have see changed information?

# in my application by url /user/change_avatar

user = self.session["user"]

user.avatar = "other_avatar_image.png"

user.put()

In session this information automatically updated?

This is a little tricky.  "user" in your example is referencing the same object as the one stored in the session.  Thus when you seen modify the avatar (even if you don't put() it) the object in the session is changed.

The tricky part is that the session object can't know that you've changed its data - it relies on you to tell it that data has changed by explicitly interacting with the session (e.g., setting a key).  So you need to tell the session that its data has been changed - otherwise the change won't be persisted.  There are a few ways to tell the session that its data has changed:

1. Set or remove any value, etc.
2. Set session.dirty = True.  In this case, I'd just do this.

~ David

[Anton Danilchenko]

I propose next improvements.

1) if I have set empty value to session - please delete this value from session.

session['user'] = None

session['user2'] = "" # also delete

session['user3'] = EmptyModel() # also delete

we need to store not empty information in sessions. Why? - I have descibe this below.

2) return not copy, but same object from session

user = session["user"]

user.avatar = "new_avatar"

# we get changed data from session, because this is one object (two links to one memory place)

# we have reload page and session have save changed information about user avatar

# we need detect changed object in session, and renew this object in memcache and in datatore

print session["user"] # show changed avatar

# delete user - this user automatically was delete from session because session have not save empty objects (I say about this above)

user.delete()

How do you think about this improvements?

[David Underhill]

On Sun, Dec 12, 2010 at 09:00, Anton Danilchenko <anton.da...@gmail.com> wrote:

I propose next improvements.

1) if I have set empty value to session - please delete this value from session.

session['user'] = None

session['user2'] = "" # also delete

session['user3'] = EmptyModel() # also delete

we need to store not empty information in sessions. Why? - I have descibe this below.

The Session is modeled after the standard Python dictionary - these values are just as valid as anything else.  

 

2) return not copy, but same object from session

user = session["user"]

user.avatar = "new_avatar"

# we get changed data from session, because this is one object (two links to one memory place)

# we have reload page and session have save changed information about user avatar

# we need detect changed object in session, and renew this object in memcache and in datatore

print session["user"] # show changed avatar

We do return a reference to the object, not a copy.  To your print statement will show the changed avatar.

 

# delete user - this user automatically was delete from session because session have not save empty objects (I say about this above)

user.delete()

This isn't possible in general - the session could be saved a value, or a value within a value, or ...

It is the user's responsibility to manage what data is in their session.  I think this is a little easier than you currently expect.

~ David

[Anton Danilchenko]

OK. Please read all my propositions above - maybe you have found some ideas is useful.

Thanks for your answers!

Regards, Anton.

[David Underhill]

The dictionary-like interface that a session provides is a familiar one and I don't plan on changing it to unset keys in new ways (e.g., when they are set to None as you proposed).

Your second proposal is already implemented - references to (not copies of) objects are returned.

~ David

[Anton Danilchenko]

Thanks!

See my new thread please.