Truncating Session Table makes site hang indefinitely

[Wade Williams]

Hi all. 

 I've got a custom session middlware built that is simply extending the session base. The only interesting thing this backend is doing is using a different session table than django's stock one -- this is for integration with a pre-existing PHP site we are in the process of migrating away from.

 Everything is working fine, except, I decided to try and truncate the sessions table in order to test something -- to my surprise, I couldn't load the site after that -- it just hung indefinitely.

 If I delete my session cookie, the site loads up no problem. 

 I'm running django 1.5a1 as we aren't scheduled for a public launch on our django code until April. Is this a 1.5 bug? Or is there something messed up with my session middleware? 

 When I try and trace where the error could be coming from, I wind up opening up the django codebase -- I don't believe there are any bugs in my middleware (especially considering it works great until you do something like truncate the sessions table).

PS I did try deleting the session table completely and re-running manage.py syncdb just incase there was some sort of relationship problem, but that doesn't appear to be the case.

Any suggestions much apprecaited. 

Thanks!

 

[Wade Williams]

Correction we're running 1.5b1 corrently.

[Russell Keith-Magee]

I can't say I've noticed any session table-related issues recently, and although it's possible I might have missed something go past, I haven't noticed any big changes to the session middleware either -- certainly nothing that sends up a red flag for the behaviour you describe. So - unfortunately, I can't really offer any advice here. 

If I were to try and debug the problem, I'd start looking into the cause of the lock. Where is the hang occurring? Is it a database-level lock trying to retrieve a row (possibly locking on a transaction?) Is the session backend looping on something? There's a loop to allocate a new session key -- if this is going into an infinite spin, it would appear as a lock. Alternatively, an operation in this loop might just be running *really* slowly; *technically*, the lock would eventually be released, it's just going to take a few minutes.

Beyond that, there's not much we can do without seeing code. You're reporting this as a problem with a custom session backend, so your first job is to prove that it isn't your own code that is the problem. If you can demonstrate the same problem with a native Django session backend, it would definitely be a release-blocking issue. If you can demonstrate that your session backend that was functional on Django 1.4 is now having problems with 1.5, then we might also have a regression that would be a release blocker -- however, it depends on the exact nature of the regression.

Yours,

Russ Magee %-)

[Wade Williams]

Thanks Russ. I too thought it might be the loop to allocate a new session key, but after studying the code and trying some forced debugs (print x) I thought it could be due to the super __init__ spinning when it couldnt find the session key in the db. I can access and insert into the DB table during the lock, and the lock never seems to end (I've run it for atleast an hour with no stop). We've built out our stuff exclusively on 1.5 so shouldnt be any 1.4 migration issues. Here is the session middleware:

from django.contrib.sessions.backends.base import SessionBase, CreateError

from django.core.exceptions import SuspiciousOperation

from django.db import IntegrityError, transaction, router

"""

 startlm/backend.py

 Custom session backend for django.

 Talks to our lm_sessions table.

 Started November 22, 2012

 By Wade Williams

 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 """

class SessionStore(SessionBase):

    """

    Implements database session store on lm_sessions table.

    Subtle differences:

    lm_sessions model is referred to as LM_Session, not Session.

    lm_session primary key field is session_id, not session_key

    So some potentially confusing details here.

    As of Now 11/24/12 the lm_sessions table does not support sessions

    expiring on time, so expire_date information is commented out.

    """

    def __init__(self, session_key=None, request=None):

        super(SessionStore, self).__init__(session_key)

        self.request = request

    def load(self):

        try:

            s = LM_Session.objects.get(

                session_id=self.session_key,

                #expire_date__gt=timezone.now()

            )

        except:

            self.create()

            return {}

        else:

            return self.decode(s.session_data)

    def exists(self, session_key):

        return LM_Session.objects.filter(session_id=session_key).exists()

    def create(self):

        while True:

            self._session_key = self._get_new_session_key()

            try:

                # Save immediately to ensure we have a unique entry in the

                # database.

                self.save(must_create=True)

            except CreateError:

                # Key wasn't unique. Try again.

                continue

            self.modified = True

            self._session_cache = {}

            return

    def save(self, must_create=False):

        """

        Saves the current session data to the database. If 'must_create' is

        True, a database error will be raised if the saving operation doesn't

        create a *new* entry (as opposed to possibly updating an existing

        entry).

        """

        # Because session middelware runs before authentication,

        # request.user isn't set until the view has been processed.

        uid = self.request.user.id

        if uid is None:

            uid = 0

        obj = LM_Session(

            session_id=self._get_or_create_session_key(),

            session_data=self.encode(self._get_session(no_load=must_create)),

            session_ip=self.request.CLIENT_IP,

            session_user_agent=self.request.META['HTTP_USER_AGENT'],

            session_user_id=uid,

            #expire_date=self.get_expiry_date()

        )

        using = router.db_for_write(LM_Session, instance=obj)

        sid = transaction.savepoint(using=using)

        try:

            obj.save(force_insert=must_create, using=using)

        except IntegrityError:

            if must_create:

                transaction.savepoint_rollback(sid, using=using)

                raise CreateError

            raise

    def delete(self, session_key=None):

        if session_key is None:

            if self.session_key is None:

                return

            session_key = self.session_key

        try:

            LM_Session.objects.get(session_id=session_key).delete()

        except LM_Session.DoesNotExist:

            pass

    @classmethod

    def clear_expired(cls):

        #Session.objects.filter(expire_date__lt=timezone.now()).delete()

        #transaction.commit_unless_managed()

        pass

# At bottom to avoid circular import

from startlm.models import LM_Session

I can add the LM_Session model and the sessionbackend we're using as well but my gut is they are not the issue at hand. 

Thanks for your help.

[Wade Williams]

ps also upgraded to 1.5.b2 with no change. 

[Wade Williams]

pps tried changing session_id to session_key to see if that solved anything with no effect.

[Russell Keith-Magee]

On Thu, Dec 20, 2012 at 9:25 AM, Wade Williams <wwil...@local-motors.com> wrote:

Thanks Russ. I too thought it might be the loop to allocate a new session key, but after studying the code and trying some forced debugs (print x) I thought it could be due to the super __init__ spinning when it couldnt find the session key in the db. I can access and insert into the DB table during the lock, and the lock never seems to end (I've run it for atleast an hour with no stop). We've built out our stuff exclusively on 1.5 so shouldnt be any 1.4 migration issues. Here is the session middleware:

 

To be clear -  it's wasn't so much migration issues I was worried about -- it was whether we've introduced a regression into Django core. Your original question was "is this a 1.5 bug?", which suggested to me that you've seen your code work on older version of Django. If that's not the case, then it seems that you're reporting "my custom session middleware doesn't work". While this is certainly a problem (for you), it's no longer pointing at a fundamental problem in Django itself that needs to be addressed before we cut a release candidate for 1.5.

As for the lock itself -- you've pointed at a number of places where the lock *isnt'*, but you haven't pointed at where the lock *is*. Is it a database lock, or a processing loop lock? Is it on retrieving the session, or creating a session? Which line of code is locking? You've said you've done some print debugging -- what lines are (and aren't) executed? 

Ultimately, you're going to need to investigate this yourself -- I (and others) can give you hints of places to look, and seeing the code will help us, but you're the one with the live test case. 

Yours,

Russ Magee %-)

[Wade Williams]

Thanks, Russ. 

 I guess i'm a little perplexed as to how exactly to debug this -- the apache log shows no errors, and trying to debug with print statements just forces the code to die due to a WSGI I/O Error. Any help that anyone can provide on debugging is much appreciated; Thanks so much for your time.

-\/\/

[Russell Keith-Magee]

Hi Wade,

If this is something you can't reproduce in testing, and print statements are causing a problem in production, don't use them -- use logging instead. Django's logging tools, which build on Python's built-in logging libraries, are designed specifically for this sort of thing. 

[1] https://docs.djangoproject.com/en/dev/topics/logging/

Yours,

Russ

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/tEDlysSgrysJ.

To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.