Duplicate session keys / uuids

94 views
Skip to first unread message

Dmitri Zagidulin

unread,
Feb 9, 2010, 12:30:24 PM2/9/10
to web2py-users
I've been experiencing some session bleed across accounts (several
instances of users crossing over into other users' sessions, and being
able to see other users' accounts). And while investigating that (by
the way, has anybody else run into this?), I've noticed that the
database in which I keep my sessions has several duplicate session
keys.

So, my main question is -- is this by design, or is something wrong?
When you store sessions in database (using session.connect - to MySQL
in this case), are there supposed to be duplicate entries with the
same uuid/session key? Would it benefit me to put in a unique
constraint on the unique_key db column?

A little more about my setup:
web2py version 1.67.2
running behind Apache/WSGI, load balanced across 3 servers. (Hence why
I'm keeping sessions in a db rather than on disk).
Sessions being stored in a MySQL db.

mdipierro

unread,
Feb 9, 2010, 9:49:31 PM2/9/10
to web2py-users
This is not normal.
Can you share more details about the setup? The amount of traffic and
the problem you encountered?

In web2py all sessions are uuid and the probablility that two uuid are
the same is null (unless these machine are one clone of the others in
which case I am not sure how the python uuid depends on the time and
ip of the machine.

Massimo

Dmitri Zagidulin

unread,
Feb 10, 2010, 10:50:02 AM2/10/10
to web2py-users
Sure.
So, this is the query that was run when session bleed was first
encountered:
select unique_key, count(unique_key) from sessions.web2py_session_init
group by unique_key having count(unique_key) > 1;
It yielded:
'0d7c9e95-ca8c-4c05-8e5f-69d058ea6510', 2
'29e390f6-f901-4312-bc5c-f42c5c4e53e7', 2
and 13 more rows like it, 15 instances of duplication in total.

The distribution in time is somewhat odd: 8 of the duplicates
instances happened on 1/25, 4 on 1/26, 1 on 1/28, 2/04 and 2/08
respectively.

The rows in question look like this:
id, locked, client_ip, created_datetime, modified_datetime, unique_key
2231, b, '192.xxx.xxx.xxx, '2010-01-25 10:53:05', '2010-01-25
10:53:05', 'd8f159bf-bdc4-4059-b4bb-307eb4880813'
2236, b, '192.xxx.xxx.xxx', '2010-01-25 10:53:05', '2010-01-25
10:53:05', 'd8f159bf-bdc4-4059-b4bb-307eb4880813' (same ip address)
...
312220, b, 'xxx.xxx.xxx.162', '2010-02-08 10:43:17', '2010-02-08
15:48:07', 'a18b5ff2-c532-4c3d-840e-880ab77d943b'
312219, b, 'xxx.xxx.xxx.68', '2010-02-08 10:43:17', '2010-02-08
10:43:17', 'a18b5ff2-c532-4c3d-840e-880ab77d943b' (different ip
addresses)

The traffic load is harder to judge, but about 16,000 session entries
in the db a day (as of 2/08).
Although now that I think about it, the most likely reason why 8 of
the duplicate session instances happened on 1/25 was because we were
doing heavy load testing (using twill) on that day. So that's likely
correlated.

The setup is: load balanced between 3 apache/wsgi servers, virtualized
via VMWare. So your comment about cloned systems and uuid gives me
pause -- the 3 servers are probably clones of each other.

Several questions:
1) I noticed that the session keys are generated by the uuid4
function. Would it help any to switch to uuid1?
2) In addition to the session key, web2py also stores the session row
id, in the session cookie. Why is that? Was there some doubt that the
session key would be unique, when you implemented that part of the
code?

3) On 2/08, the database shows only one instance of duplicate session
keys. However, there were about 20 reports of users' sessions being
crossed over (and users seeing other users' accounts when logged in).
So those two things don't seem to be directly correlated. We're still
trying to diagnose this session bleed issue, rolling back revisions
and hoping to see if it was a code issue or an environment leak
somehow.

Thanks!

Dmitri Zagidulin

unread,
Feb 10, 2010, 3:51:06 PM2/10/10
to web2py-users
More details.

The session bleed issue is happening because of a code change (not a
random environment leak). We've replicated it twice -- applied the 2
code changes (described below), immediately get complaints about
session bleed, roll back the changes, everything's fine.

The problem is, I can't see how the changes in question would lead to
messed up sessions. They are:

1) Cache-control statements. Specifically, in one controller, inserted
the following line:
response.headers['Cache-Control'] = 'max-age=3600, must-revalidate'
and in another,
response.headers['Cache-Control'] = 'store, no-cache, must-revalidate'

2) A change in where to redirect the user after login.
was (in db.py):
auth.settings.login_next = 'https://appurl/user_home'

changed to:
auth.settings.login_next = 'http://appurl/user_home'

But how could this be?

Why would changing the browser cache-control headers, or redirecting
to an http url instead of https, cause one user to get another user's
session (and see the other user's account, etc)?

mdipierro

unread,
Feb 10, 2010, 8:18:09 PM2/10/10
to web2py-users
I think you found the problem and yes it should be considered a defect
in web2py:

http://stackoverflow.com/questions/1785503/when-should-i-use-uuid-uuid1-vs-uuid-uuid4-in-python

We should replace all uuid4 with uuid1 else the PRNGs will generate
collisions because they ARE NOT independent on cloned virtual servers.

Any comment?

Alexandre Andrade

unread,
Feb 10, 2010, 8:32:19 PM2/10/10
to web...@googlegroups.com
A 'non-error' aproach is better than a 'probablily non-error' one.


2010/2/10 mdipierro <mdip...@cs.depaul.edu>
--
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to web2py+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/web2py?hl=en.




--
Atenciosamente

--
=========================
Alexandre Andrade
Hipercenter.com

mdipierro

unread,
Feb 10, 2010, 8:44:03 PM2/10/10
to web2py-users
The issue here is that replacing uuid4 with uiid no longer generates
collisions in the case discussed above but it may be possible to guess
keys and steal a session, although it is unlikely the attacker can get
enough info to do so.

Perhaps we should just hmac uuid1 with uuid4 and user our own
function.

On Feb 10, 7:32 pm, Alexandre Andrade <alexandrema...@gmail.com>
wrote:


> A 'non-error' aproach is better than a 'probablily non-error' one.
>

> 2010/2/10 mdipierro <mdipie...@cs.depaul.edu>


>
>
>
> > I think you found the problem and yes it should be considered a defect
> > in web2py:
>

> >http://stackoverflow.com/questions/1785503/when-should-i-use-uuid-uui...

> > web2py+un...@googlegroups.com<web2py%2Bunsu...@googlegroups.com>

mdipierro

unread,
Feb 15, 2010, 1:07:05 PM2/15/10
to web2py-users
The web2py in trunk has a new uuid function that combines uuid4 (it is
random) and uuid1 (takes into account mac address and time). It should
void uuid conflicts when running multiple vps instances clones of each
other. You may want to give it a try.

Massimo

On Feb 10, 2:51 pm, Dmitri Zagidulin <dzagidu...@gmail.com> wrote:

Dmitri Zagidulin

unread,
Feb 15, 2010, 1:47:34 PM2/15/10
to web2py-users
Do I have to make code changes, to use the new function, or just
update to trunk?

(Also, do you by any chance have any insight into the question above,
on how could either cache control or changing the redirect destination
after login, cause session crossovers and bleed?)

mdipierro

unread,
Feb 15, 2010, 2:10:05 PM2/15/10
to web2py-users
> Why would changing the browser cache-control headers, or redirecting
> to an http url instead of https, cause one user to get another user's
> session (and see the other user's account, etc)?

this is impossible. What may have caused the problem is that switching
form http to https cased the client not to send the session cookie
back. The server created a new session. Because your machines are
clones of each other, the new session could have had the same uuid as
the session of a different user created by a different machine. So in
effect the two users shared one session it. The http/https switch only
made this more likely. The fix in trunk will prevent this form
happening again.

For everybody else. This is only a problem when you have cloned
machines (indentical PRNG) behind a load balancer and it will be fixed
in 1.75.2.

Reply all
Reply to author
Forward
0 new messages