does anyone have a server-side session library other than pyramid_redis_sessions ?

[Jonathan Vanasco]

I forked `pyramid_redis_sessions` into a new project because my needs changed.

i ended up inheriting a design flaw and need to refactor it out.  I'm hoping to look at other server-side session systems for inspiration.

my issue is that not having a session_id cookie, or having an invalid one, will automatically generate a placeholder session in redis.  i need to refactor my code so that session_ids are only created if actually used.  (i have a handful of ideas on how to do this, but if something works... inspiration would save me time).

this isn't an issue for normal usage, but i got hit by a spider the other day that ignored cookies and created over 30k stale sessions.  this will pose a problem for me in the future and i'd like to address it (pagecount is growing, session and database-cache are in redis running LRU mode).

[Jeff Dairiki]

I have written a custom session implementation which is a hybrid between cookie-storage and redis-storage.  If the session dict is JSON-serializable and is not too large, then the session is just stored directly in a cookie.  Otherwise, the session data is pickled and stored in redis.  In the latter case just a session id (as well as, in our case, a few other keys having to do with user authentication) is stored in the session cookie.

I did this, in part to solve the zealous spider problem you describe.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/20766195-47b0-46eb-a82f-06c331ce9ec8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Vanasco]

That sounds pretty smart.

So I assume you defer creating a null session until it's used?

[Jeff Dairiki]

On Fri, Feb 3, 2017 at 6:37 PM, Jonathan Vanasco <jona...@findmeon.com> wro

So I assume you defer creating a null session until it's used?

Actually, no.  Pretty much every page includes a CSRF token somewhere, and thus require a session.  However these simple sessions are stored entirely in the session cookie, so no server-side storage is required.

[Jonathan Vanasco]

On Sunday, February 5, 2017 at 4:16:29 PM UTC-5, Jeff Dairiki wrote:

Actually, no.  Pretty much every page includes a CSRF token somewhere, and thus require a session.  However these simple sessions are stored entirely in the session cookie, so no server-side storage is required.

Sorry, I meant to describe a placeholder session in redis.  At some point you decide a session id is needed for redis.

[Mikko Ohtamaa]

this isn't an issue for normal usage, but i got hit by a spider the other day that ignored cookies and created over 30k stale sessions.  this will pose a problem for me in the future and i'd like to address it (pagecount is growing, session and database-cache are in redis running LRU mode).

30k sessions should not be an issue yet. I had 400k sessions and no issues there. Redis should be very high amount of keys easily.

My suggestion is that

* Have 24 hours or less expiration time for anonymous sessions

* Upgrade the session expiration time when the user logs in, so that logged in sessions are not forgotten

Also I am pretty sure one can configure pyramid_redis_session not create session until you save something there. Though not 100% sure here. However I am saving some stats per user when they come to the site for the first time (affiliate, etc.)

My 2c (Euro),

Mikko

--

Mikko Ohtamaa
http://opensourcehacker.com
http://twitter.com/moo9000

[Jonathan Vanasco]

On Monday, February 6, 2017 at 3:21:45 PM UTC-5, Mikko Ohtamaa wrote:

30k sessions should not be an issue yet. I had 400k sessions and no issues there. Redis should be very high amount of keys easily.

We don't have a problem now, but will in the future.  Right now we're forecasting that 500k "bad" bot sessions will start to be an issue for us.  

Redis can handle millions of keys fine, and could handle 10MM of these sessions on their own.  But each "bot session" corresponds to a single page view, which has multiple elements that are backed by our Redis caches as well (partial html fragments, object caching, etc).  Based on current usage, at around 500k "sessions" we will begin to max out Redis in terms of data-size (not number of keys).

Since this is a "bad bot", all these requests happen rather quickly and through a bunch of distributed IPs (i think it's a proxy fronted botnet that is indexing) and don't respect rate-limit requests.  Blocking them at the firewall is a lot of work too, and then they will pop up again.

We operate Redis in a LRU mode, so valid data will be pushed out to make room for bot sessions -- the oldest data will largely be the sessions of actual users.  The Redis node with sessions is shared within a cluster -- so a botnet hitting 10 application servers at the same time is hitting Redis 10x at the same time.  This becomes a really annoying performance problem fast.

We don't currently use TTLs in Redis.  That was recently removed for better performance and more room -- redis maintains the TTLs separately, so you end up having 2 lookups and payloads within redis.  This isn't noticeable under most conditions, but it causes delays during peak load times (and wastes a chunk of space).  We recently shifted the timeout to be entirely within the payload (python), and eliminate the EXPIRY calls.  

Unfortunately pyramid_redis_sessions can't support a "lazy" create without substantial rewriting.  The current design creates a placeholder session in redis when a new session is accessed (or an existing one is invalidated).  Our fork is in a decent place to get around this, because we already defer persist/refresh into callbacks.

Thinking out-loud, it may make sense to support multiple expiration policies -- put a very short TTL on unauthenticated sessions, and keep the authenticated ones as a strict LRU with 24hr sessions.

[Bert JW Regeer]

On Feb 3, 2017, at 14:07, Jonathan Vanasco <jona...@findmeon.com> wrote:

I forked `pyramid_redis_sessions` into a new project because my needs changed.

i ended up inheriting a design flaw and need to refactor it out.  I'm hoping to look at other server-side session systems for inspiration.

my issue is that not having a session_id cookie, or having an invalid one, will automatically generate a placeholder session in redis.  i need to refactor my code so that session_ids are only created if actually used.  (i have a handful of ideas on how to do this, but if something works... inspiration would save me time).

Don’t touch request.session…

Pyramid doesn’t actually call the session factory unless you ask for it from request.session.

Could create a pass-through session factory that won’t call into redis session unless a key is being set (access thus won’t create a session, so you can do things like ‘key’ in blah).

As it currently stands, it doesn’t look like redis sessions plays well with transactions, based upon a quick glance at the code it looks like it refreshes/persists data the moment you set it.

Warehouse actually disables the session entirely if it is a stateless request that shouldn’t be touching the session which is a neat idea too.

this isn't an issue for normal usage, but i got hit by a spider the other day that ignored cookies and created over 30k stale sessions.  this will pose a problem for me in the future and i'd like to address it (pagecount is growing, session and database-cache are in redis running LRU mode).

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.

[Jonathan Vanasco]

On Monday, February 6, 2017 at 6:18:02 PM UTC-5, Bert JW Regeer wrote:

Don’t touch request.session…

I wish!  We have to do a read for login status and some other things.  Pushing that into a secondary session was an option, but that's gone.

Could create a pass-through session factory that won’t call into redis session unless a key is being set (access thus won’t create a session, so you can do things like ‘key’ in blah).

 

That is a brilliant idea!

 

As it currently stands, it doesn’t look like redis sessions plays well with transactions,

IMHO, that is a good thing.  Sessions can store transactional-dependent data via multiple writes and a callback.  A lot of times we store data in a session that is important to have regardless of transaction status -- like 3rd party calls that are not transaction aware or the last-5 page views.

based upon a quick glance at the code it looks like it refreshes/persists data the moment you set it.

From what I recall, it serializes the session and calls SET on every writes and sends an EXPIRE on most other operations.  The amount of traffic was hurting our installation.  The fork I wrote (pyramid_session_redis) registers a callback to call set/expire only once.  

I've nearly got a solution in place for this.  It was harder to write because I am avoiding a rewrite.

[Jeff Dairiki]

Yes, that's right.  The session id for redis is not created until it is decided that the session will be stored in redis. (This happens when the session dict is not JSON-serializable, or when the size of the JSON serialization exceeds a configured limit.)  At least with our usage, most sessions, including those created by bots do not hit redis at all.

[Mike Orr]

This is in the regular 'pyramid_redis_sessions', or this is in a fork?

[Jeff Dairiki]

This is custom (private) code.  It is not based on pyramid_redis_sessions at all.  It does, however, extend the standard pyramid SignedCookieSessionFactory for session cookie management, and dogpile.cache is used for handling the server-side (redis) data.

If there's real interest in it, I can look into splitting it into it's own library.

[Jonathan Vanasco]

On Wednesday, February 8, 2017 at 5:54:57 PM UTC-5, Jeff Dairiki wrote:

If there's real interest in it, I can look into splitting it into it's own library.

I'd be interested in seeing it.  I don't know if we'd be able to use it, as we need to force some things to be stored server side (even if a session is encrypted, there is still a bag of worms called "compliance" that I do not want to touch)

I've got a new version of our `pyramid_session_redis` testing on staging right now.  if it works, I'll try a production release and go to pypi.  I'm not happy with the code/design because I tried to maintain compatibility (then decided to break it), but the features work as needed and a few dozen new tests pass.

I modeled a lot of stuff after dogpile (the payload tracks an api version, a class instance is used to track null ids) and tracked a timestamp into the payload so updating the timeout can be deferred until a threshold is reached.  The redis traffic on our staging server is already a lot lower.