How to get more predictable caching behavior - how to store sessions in memcached
martin.grotzke
I know that this topic is rather burdened, as it was said often enough
that memcached never was created to be used like a reliable datastore.
Still, there are users interested in some kind of reliability, users
that want to store items in memcached and be "sure" that these items
can be pulled from memcached as long as they are not expired.
I read the following on memcached's memory management:
"Memcached has two separate memory management strategies:
- On read, if a key is past its expiry time, return NOT FOUND.
- On write, choose an appropriate slab class for the value; if it's
full, replace the oldest-used (read or written) key with the new one.
Note that the second strategy, LRU eviction, does not check the expiry
time at all." (from "peeping into memcached", [1])
I also found "Slabs, Pages, Chunks and Memcached" ([2]) a really good
explanation of memcached's memory model.
Having this as background, I wonder how it would be possible to get
more predictability regarding the availability of cached items.
Asume that I want to store sessions in memcached. How could I run
memcached so that I can be sure that my sessions are available in
memcached when I try to "get" them? Additionally asume, that I expect
to have 1000 sessions at a time in max in one memcached node (and that
I can control/limit this in my application). Another asumption is,
that sessions are between 50kb and 200 kb.
The question now is how do I have to run memcached to "store" these
sessions in memcached?
Would it be an option to run memcached with a minimum slab size of
200kb. Then I would know that for each session a 200kb chunk is used.
When I have 1000 session between 50kb and 200kb this should take 200mb
in total. When I run memcached with more than 200mb memory, could I be
sure, that the sessions are alive as long as they are not expired?
What do you think about this?
Cheers,
Martin
[1] http://blog.evanweaver.com/articles/2009/04/20/peeping-into-memcached/
[2] http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/
Ren
http://dormando.livejournal.com/495593.html
Jared
On Mar 12, 9:02 pm, "martin.grotzke" <martin.grot...@googlemail.com>
wrote:
Martin Grotzke
Martin Grotzke
http://www.javakaffee.de/blog/
Brian Moon
and won't with memcached. That is not its job. It will never be its job.
Perhaps when storage engines are done, maybe. But then you won't get the
performance that you get with memcached. There is a trade off for
performance.
Brian.
--------
http://brian.moonspot.net/
Les Mikesell
> With my question "how do I have to run memcached to 'store' these
> sessions in memcached" I was not referring to a general approach, but I
> was referring to the concrete memcached options (e.g. -n 204800 for
> 200kb slabs) to use.
>
> The post you mentioned is very high level and does not answer my
> question. For this you should go into a little more depth.
Some details here:
http://dev.mysql.com/doc/refman/5.0/en/ha-memcached-using-memory.html
But it's still not very clear how much extra you need to make sure that
the hash to a server/slab will always find free space instead of
evicting something even though space is available elsewhere.
--
Les Mikesell
lesmi...@gmail.com
Martin Grotzke
Marc Bollinger
'store' these sessions in memcached," is not a concrete question. It's
wibbly wobbly at best to try and achieve this behavior, and, "You
can't," _is_ concrete in that there is no way to do this in a
mathematically provable way. The best you're going to get is something
that anecdotally works, contingent on the existence of roughly
homogeneous object sizes, and that you're allocating enough memory to
memcached. The scheme you described above (maybe tweak the growth
factor downward to taste?) will probably work, but that assumes a
limit of 1000 users, which is unrealistic for the scale that memcached
was really designed for in the first place.
- Marc
Martin Grotzke
On 3/12/2010 5:10 PM, Martin Grotzke wrote:Some details here:
With my question "how do I have to run memcached to 'store' these
sessions in memcached" I was not referring to a general approach, but I
was referring to the concrete memcached options (e.g. -n 204800 for
200kb slabs) to use.
The post you mentioned is very high level and does not answer my
question. For this you should go into a little more depth.
http://dev.mysql.com/doc/refman/5.0/en/ha-memcached-using-memory.html
When you start to store data into the cache, memcached does not allocate the memory for the data on an item by item basis. Instead, a slab allocation is used to optimize memory usage and prevent memory fragmentation when information expires from the cache.
With slab allocation, memory is reserved in blocks of 1MB. The slab is divided up into a number of blocks of equal size.
When you try to store a value into the cache, memcached checks the size of the value that you are adding to the cache and determines which slab contains the right size allocation for the item. If a slab with the item size already exists, the item is written to the block within the slab.
If the new item is bigger than the size of any existing blocks,
then a new slab is created, divided up into blocks of a suitable size.
If an existing slab with the right block size already exists,
but there are no free blocks, a new slab is created.
But it's still not very clear how much extra you need to make sure that the hash to a server/slab will always find free space instead of evicting something even though space is available elsewhere.
--
Les Mikesell
lesmi...@gmail.com
Martin Grotzke
dormando
>
> I know that this topic is rather burdened, as it was said often enough
> that memcached never was created to be used like a reliable datastore.
> Still, there are users interested in some kind of reliability, users
> that want to store items in memcached and be "sure" that these items
> can be pulled from memcached as long as they are not expired.
To be clear; I'm going to answer your question to the best of my ability,
but I want to communicate in a more clear manner what other folks are
trying to say.
In this use case, we can answer your question to the tune of "how can you
make memcached most efficient for holding active sessions". You cannot
guarantee that data will not be lost if you will ever have more sessions
than memory, minus overhead. The post I wrote is answering a question that
you're not asking, but we all wish folks would ask:
- How do I use memcached to make my sessions more efficient?
Instead you're asking "how do I store them in memcached anyway", which is
drawing some ire. I apologize for the bluntness the list has been giving
you on this. It's generally preferred that memcached is used as a method
of improving performance of another system.
For instance, the speed of memcachedb, tokyotyrant, or anything that hits
disk ever, will have its performance slow to complete garbage once the
set of active data is larger than the memory allocated to the service.
This is true for everything everywhere ever. TT, MySQL, CouchDB, anything.
Memcached provides a flexible pool of memory you can put in front of
something like this to scale to meet the data requirements of your active
data set.
In this case "active dataset" probably means sessions that would be looked
at within the same few minutes.
> I read the following on memcached's memory management:
> "Memcached has two separate memory management strategies:
> - On read, if a key is past its expiry time, return NOT FOUND.
> - On write, choose an appropriate slab class for the value; if it's
> full, replace the oldest-used (read or written) key with the new one.
> Note that the second strategy, LRU eviction, does not check the expiry
> time at all." (from "peeping into memcached", [1])
This is close but not true.
- On read, if a key is past its expiry time, return its memory to the slab
pool and return NOT FOUND
- On write, try to allocate new memory:
* From the slab's pool of free memory.
* ... and if the slab is full, add another megabyte from the main pool
* ... if the main pool is fully assigned out to slab classes, find
something to evict.
* for that slab class, look at the LRU tail (oldest items in the cache)
* walk up the tail looking for *items that are already expired*
* after 50 items have been examined, if no expired items are found,
* walk the list again, and *expire* the first thing you find
(I'm leaving out refcount locks for sake of this discussion)
So what it finds is the absolute oldest data in your cache and chucks it,
regardless of expired or not, but it at least makes an attempt to find an
expired item.
Think about how this aligns to a session store, however:
- I load your web page. This causes my session to be loaded. This will
cause my session item to be bumped to the front of the LRU list (though an
item can only be bumped once per minute, to avoid constant shuffling).
- This also may cause my session to freshen itself, as you've updated my
"last_visited" column and ->set the session back into memcached. This also
causes my session to land at the head of the LRU.
- I stop visiting your website. My session will time out after 60 minutes.
- Other people visit the website. Slowly my session creeps down the LRU
toward the tail.
Now, it should be obvious that if a user session has reached a point where
it would be evicted early, it is because you did not have enough memory to
store *all active sessions anyway*. The odds of it evicting someone who
has visited your site *after* me are highly unlikely. The longer I stay
off the site, the higher the odds of it being evicted early due to lack of
memory.
This does mean, by way of painfully describing how an LRU works, that the
odds of you finding sessions in memcached which have not been expired, but
are being evicted from the LRU earlier than expired sessions, is very
unlikely.
I'll continue with the caveat in a moment.
> I also found "Slabs, Pages, Chunks and Memcached" ([2]) a really good
> explanation of memcached's memory model.
>
> Having this as background, I wonder how it would be possible to get
> more predictability regarding the availability of cached items.
>
> Asume that I want to store sessions in memcached. How could I run
> memcached so that I can be sure that my sessions are available in
> memcached when I try to "get" them? Additionally asume, that I expect
> to have 1000 sessions at a time in max in one memcached node (and that
> I can control/limit this in my application). Another asumption is,
> that sessions are between 50kb and 200 kb.
>
> The question now is how do I have to run memcached to "store" these
> sessions in memcached?
>
> Would it be an option to run memcached with a minimum slab size of
> 200kb. Then I would know that for each session a 200kb chunk is used.
> When I have 1000 session between 50kb and 200kb this should take 200mb
> in total. When I run memcached with more than 200mb memory, could I be
> sure, that the sessions are alive as long as they are not expired?
The caveat is that memcached has one LRU per slab class.
So, lets say your traffic ends up looking like:
- For the first 10,000 sessions, they are all 200 kilobytes. This ends up
having memcached allocate all of its slab memory toward something that
will fit 200k items.
- You get linked from the frontpage of digg.com and suddenly you have a
bunch of n00bass users hitting your site. They have smaller sessions since
they are newbies. 10k items.
- Memcached has only reserved 1 megabyte toward 10k items. So now all of
your newbies share a 1 megabyte store for sessions, instead of 200
megabytes.
If you set the minimum slab size to 200k, you unify the memory so the
largest pool of memory is always available for your users. However, you
drastically raise the memory overhead for users with 10k sessions. 90%+
overhead. In reality when you start a memcached instance and throw traffic
at it, sessions will either be very close together in size, or vary by
some usable distribution.
If it ends up changing over time, the only present workaround is to
restart memcached. This sucks since it'll end up logging your users out.
So this is why we recommend that folks store sessions in something disk
backed and/or centralized. I don't view an RDBMS as over-engineered for a
problem... if you already have one just create a table on it and use it
for fuck's sake. Cache the hell out of it so you barely add any load to
the system and be done with it. If you have to think this hard to store
sessions you've probably already lost. Brian Moon's suggestion of having
sessions split between "user state" and "fiddly logged in user stuff" is
also a good suggestion.
However the slab out of balance thing is a real fault of ours. It's a
project on my plate to have automated slab rebalancing done in some usable
fashion within the next several weeks. This means that if a slab is out of
memory and under pressure, memcached will decide if it can pull memory
from another slab class to satisfy that need. As the size of your items
change over time, it will thus try to compensate.
This still doesn't mean you should rely on it holding all of your sessions
(your sysadmins will hate you for never allowing them to upgrade
memcached, or the host OS, or reboot or add servers or anything), but it
will help with the predictability of how much benefit you will receive,
and reduce maintenance effort.
-Dormando
dormando
> http://dev.mysql.com/doc/refman/5.0/en/ha-memcached-using-memory.html
>
> thanx for this link. Some details in the text are confusing to me. It says:
>
> When you start to store data into the cache,�memcached�does not allocate the memory for the data on an item by item basis. Instead, a slab allocation is used to optimize memory usage and
> prevent memory fragmentation when information expires from the cache.
>
>
> With slab allocation, memory is reserved in blocks of 1MB. The slab is divided up into a number of blocks of equal size.
>
> Ok, blocks of equal size, all blocks have 1 MB (as said in the sentence before).
>
> When you try to store a value into the cache,�memcached�checks the size of the value that you are adding to the cache and determines which slab contains the right size allocation for the item.
> If a slab with the item size already exists, the item is written to the block within the slab.
>
> "written to the block within the slab" sounds, as if there's one block for one slab?
> �
>
>
> If the new item is bigger than the size of any existing blocks,
>
> I thought all blocks are 1MB in their size? Should this be "of any existing slab"?
> �
> then a new slab is created, divided up into blocks of a suitable size.
>
> Again, I thought blocks are 1MB, then what is a suitable size here?
> �
> If an existing slab with the right block size already exists,
>
> Confusing again.
> �
> but there are no free blocks, a new slab is created.
>
>
> In the second part of this documentation, the terms "page" and "chunk" are used, but not related to "block". "block" is not used at all in the second part. Can you clarify the meaning of block in this
> context and create a link to slab, page and chunk?
>
> Btw, I found "Slabs, Pages, Chunks and Memcached" ([1]) really well written and easy to understand. Would say this explanation is complete?
The memory allocation is a bit more subtle... but it's hard to explain and
doesn't really affect anyone.
Urr... I'll give it a shot.
./memcached -m 128
^ means memcached can use up to 128 megabytes of memory for item storage.
Now lets say you store items that will fit in a slab class of "128", which
means 128 bytes for the key + flags + CAS + value.
The maximum item size is (by default) 1 megabyte. This ends up being the
ceiling for how big a slab "page" can be.
8192 128 byte items will fit inside the slab "page" limit of 1mb.
So now a slab page of 1mb is allocated, and split up into 8192 "chunks".
Each chunk can store a single item.
Slabs grow at a default factor of 1.20 or whatever -f is. So the next slab
class after 128 bytes will be 154 bytes (rounding up). (note I don't
actually recall offhand if it rounds up or down :P)
154 bytes is not evenly divisible into 1048576. You end up with
6808.93 chunks. So instead memcached allocates a page of 1048432 bytes,
providing 6,808 chunks.
This is slightly smaller than 1mb! So as your chunks grow, they
don't allocate exactly a megabyte from the main pool of "128 megabytes",
then split that into chunks. Memcached attempts to leave the little scraps
of memory in the main pool in hopes that they'll add up to an extra page
down the line, rather than be thrown out as overhead when if a slab class
were to allocate a 1mb page.
So in memcached, a slab "page" is "however many chunks of this size will
fit into 1mb", a chunk is "how many chunks will fit into that page". The
slab growth factor determines how many slab classes exist.
I'm gonna turn this into a wiki entry in a few days... been slowly
whittling away at revising the whole thing.
-Dormando
dormando
To be most accurate, it is "how many chunks will fit into the max item
size, which by default is 1mb". The page size being == to the max item
size is just due to how the slabbing algorithm works. It creates slab
classes between a minimum and a maximum. So the maximum ends up being the
item size limit.
I can see this changing in the future, where we have a "max item size" of
whatever, and a "page size" of 1mb, then larger items are made up of
concatenated smaller pages or individual malloc's.
Les Mikesell
>>
> To be most accurate, it is "how many chunks will fit into the max item
> size, which by default is 1mb". The page size being == to the max item
> size is just due to how the slabbing algorithm works. It creates slab
> classes between a minimum and a maximum. So the maximum ends up being the
> item size limit.
>
> I can see this changing in the future, where we have a "max item size" of
> whatever, and a "page size" of 1mb, then larger items are made up of
> concatenated smaller pages or individual malloc's.
So what happens when a key is repeatedly written and it grows a bit each time?
I had trouble with that long ago with a berkeleydb version that I think was
eventually fixed. As things work now, if the new storage has to move to a
larger block, is the old space immediately freed?
--
Les Mikesell
lesmi...@gmail.com
Dustin
On Mar 12, 6:10 pm, Les Mikesell <lesmikes...@gmail.com> wrote:
> So what happens when a key is repeatedly written and it grows a bit each time?
> I had trouble with that long ago with a berkeleydb version that I think was
> eventually fixed. As things work now, if the new storage has to move to a
> larger block, is the old space immediately freed?
Every write is to a new block, then the previous is made available
for the next write. Size isn't relevant for this case.
Carlos Alvarez
<martin....@googlemail.com> wrote:
> Hi Brian,
> you're making a very clear point. However it would be nice if you'd provide
> concrete answers to concrete questions. I want to get a better understanding
> of memcached's memory model and I'm thankful for any help I'm getting here
> on this list. If my intro was not supporting this please forgive me...
> Cheers,
> Martin
Well, you asked "how do I ..." and the answer was "you can't". It
sounds quite concrete to me. :-)
Anyway, if you want to try (you'll face risk and your solution will be
error prone, don't forget that) I remember lurking around the code and
seeing an option of 'no evictions': ie when there is not enough memory
the set/add fails. I don't know if it this option is fully functional,
but the code is there.
if (settings.evict_to_free == 0) {
itemstats[id].outofmemory++;
return NULL;
}
In this case, when you run out of memory to store sessions, you'll
notice in the 'overflow' session and not in a older one (I would
prefer that).
Anyway, remember that evictions is not the only cause of not being
able to retrieve previously stored items.
Carlos.
Martin Grotzke
- On read, if a key is past its expiry time, return its memory to the slab
pool and return NOT FOUND
- On write, try to allocate new memory:
* From the slab's pool of free memory.
* ... and if the slab is full, add another megabyte from the main pool
* ... if the main pool is fully assigned out to slab classes, find
something to evict.
* for that slab class, look at the LRU tail (oldest items in the cache)
* walk up the tail looking for *items that are already expired*
* after 50 items have been examined, if no expired items are found,
* walk the list again, and *expire* the first thing you find
(I'm leaving out refcount locks for sake of this discussion)
This does mean, by way of painfully describing how an LRU works, that the
odds of you finding sessions in memcached which have not been expired, but
are being evicted from the LRU earlier than expired sessions, is very
unlikely.
So, lets say your traffic ends up looking like:
- For the first 10,000 sessions, they are all 200 kilobytes. This ends up
having memcached allocate all of its slab memory toward something that
will fit 200k items.
- You get linked from the frontpage of digg.com and suddenly you have a
bunch of n00bass users hitting your site. They have smaller sessions since
they are newbies. 10k items.
- Memcached has only reserved 1 megabyte toward 10k items. So now all of
your newbies share a 1 megabyte store for sessions, instead of 200
megabytes.
If you set the minimum slab size to 200k, you unify the memory so the
largest pool of memory is always available for your users. However, you
drastically raise the memory overhead for users with 10k sessions. 90%+
overhead.
In reality when you start a memcached instance and throw traffic
at it, sessions will either be very close together in size, or vary by
some usable distribution.
If it ends up changing over time, the only present workaround is to
restart memcached. This sucks since it'll end up logging your users out.
However the slab out of balance thing is a real fault of ours. It's a
project on my plate to have automated slab rebalancing done in some usable
fashion within the next several weeks. This means that if a slab is out of
memory and under pressure, memcached will decide if it can pull memory
from another slab class to satisfy that need. As the size of your items
change over time, it will thus try to compensate.
Martin Grotzke
dormando
You really don't want to mess with this value. It will bring you the
absolute opposite results of what you expect. Memcached does this search
while holding a global mutex lock, so no other threads are able to access
the allocator at the same time. It must be very fast, and most people will
just crank it and grind the system to a halt, so I'm unsure if we will
make this configurable.
The number's just in a couple places in items.c if you wish to recompile
it and test, but again that's absolutely not a good idea if you want it to
perform.
�
> That's right for the normal case. However, for the memcached-session-manager I just implemented a feature so that sessions are only sent to memcached for backup if session data was modified. To prevent
> expiration of the session in memcached, a background thread is updating sessions in memcached that would expire ~20 seconds later in memcached (if there would be some kind of "touch" operation I'd just use
> that one). When they are updated, they are set with an expiration of the remaining expiration time in tomcat. This would introduce the issue, that the update would push them to the head of the LRU, but
> their expiration might be e.g. only 5 minutes. So they would be expired but won't be reached when the 50 items are checked for expiration.
Are sessions modified on every hit? or just fetched on every hit? I don't
see how this would be different since active sessions are still getting
put to the front of the LRU. When you modify a session isn't the
expiration time of that session extended usually? Or do users get logged
out after they've run out of their 5 free hours of AOL?
Honestly, if you have a background thread that can already find sessions
when they're about to expire, why not issue a DELETE to memcached when
they do expire? Or even a GET after they expire :)
> That's true, this is the drawback that I'd have to accept.
>
> Do you see other disadvantages with this approach (e.g. performance wise)?
Shouldn't be any particular performance disadvantage, other than serverely
reducing the effectiveness of your cache and caching fewer items overall
:P
> Yes, for this I (the memcached-session-manager) should provide some stats on size distribution.
> Or does memcached already provide this information via its stats?
`stats items` and `stats slabs` provide many statistics on individual
slabs. You can monitor this to see if a particular slab size is having a
higher number of evictions and doesn't have enough slabs assigned to it.
As well as how many items are in each slab and all that junk.
�
> In my case this is not that much an issue�(users won't get logged out), as sessions are served from local memory. Session are only in memcached for the purpose of session failover. So this restart could be
> done when operations could be *sure* that no tomcat will die.
Tomcat sounds like such a pisser :P Even with your backup thing I'd
probably still add an option to allow it to journal to a database, and I
say this knowing how to get every last ounce of efficiency out of
memcached.
-Dormando
Adam Lee
Peter J. Holzer
> Now, it should be obvious that if a user session has reached a point where
> it would be evicted early, it is because you did not have enough memory to
> store *all active sessions anyway*. The odds of it evicting someone who
> has visited your site *after* me are highly unlikely. The longer I stay
> off the site, the higher the odds of it being evicted early due to lack of
> memory.
>
> This does mean, by way of painfully describing how an LRU works, that the
> odds of you finding sessions in memcached which have not been expired, but
> are being evicted from the LRU earlier than expired sessions, is very
> unlikely.
> The caveat is that memcached has one LRU per slab class.
>
> So, lets say your traffic ends up looking like:
>
> - For the first 10,000 sessions, they are all 200 kilobytes. This ends up
> having memcached allocate all of its slab memory toward something that
> will fit 200k items.
> - You get linked from the frontpage of digg.com and suddenly you have a
> bunch of n00bass users hitting your site. They have smaller sessions since
> they are newbies. 10k items.
> - Memcached has only reserved 1 megabyte toward 10k items. So now all of
> your newbies share a 1 megabyte store for sessions, instead of 200
> megabytes.
There's another caveat (I think Martin may have been referring to this
scenario, but he wasn't very clear):
Suppose you have two kinds of entries in your memcached, with different
expire times. For example, in addition to your sessions with 3600s, you
have some alert box with an expiry time of 60s. By chance,
both items are approximately the same size and occupy the same slab
class(es).
You have enough memory to keep all sessions for 3600 seconds and enough
memory to keep all alert boxes for 60 seconds. But you don't have enough
memory to keep all alert boxes for 3600 seconds (why should you, they expire
after 60 seconds).
Now, when you walk the LRU chain, the search for expired items will only
return expired alert boxes which are about as old as your oldest session.
As soon as there are 50 (not yet expired) sessions older than the oldest
(expired) alert box, you will evict a session although you still have a
lot of expired alert boxes which you could reuse.
The only workaround for this problem I can see is to use different
memcached servers for items of (wildly) different expiration times.
> However the slab out of balance thing is a real fault of ours. It's a
> project on my plate to have automated slab rebalancing done in some usable
> fashion within the next several weeks. This means that if a slab is out of
> memory and under pressure, memcached will decide if it can pull memory
> from another slab class to satisfy that need. As the size of your items
> change over time, it will thus try to compensate.
That's good to hear.
hp
--
_ | Peter J. Holzer | Openmoko has already embedded
|_|_) | Sysadmin WSR | voting system.
| | | h...@hjp.at | Named "If you want it -- write it"
__/ | http://www.hjp.at/ | -- Ilja O. on comm...@lists.openmoko.org
Martin Grotzke
> That's right for the normal case. However, for the memcached-session-manager I just implemented a feature so that sessions are only sent to memcached for backup if session data was modified. To preventAre sessions modified on every hit? or just fetched on every hit?
> expiration of the session in memcached, a background thread is updating sessions in memcached that would expire ~20 seconds later in memcached (if there would be some kind of "touch" operation I'd just use
> that one). When they are updated, they are set with an expiration of the remaining expiration time in tomcat. This would introduce the issue, that the update would push them to the head of the LRU, but
> their expiration might be e.g. only 5 minutes. So they would be expired but won't be reached when the 50 items are checked for expiration.
I don't
see how this would be different since active sessions are still getting
put to the front of the LRU.
When you modify a session isn't the
expiration time of that session extended usually?
Honestly, if you have a background thread that can already find sessions
when they're about to expire, why not issue a DELETE to memcached when
they do expire? Or even a GET after they expire :)
> In my case this is not that much an issue (users won't get logged out), as sessions are served from local memory. Session are only in memcached for the purpose of session failover. So this restart could beTomcat sounds like such a pisser :P Even with your backup thing I'd
> done when operations could be *sure* that no tomcat will die.
probably still add an option to allow it to journal to a database, and I
say this knowing how to get every last ounce of efficiency out of
memcached.
-Dormando
Adam Lee
with "what if?" scenarios for memcached (particularly in regard to
node failures and data integrity) and, while they're technically
possible, they very, very rarely happen in the wild. for this one,
i'll just say that memcached does a great job with its slab allocation
and, unless you're running with way too little memory, you'll not very
often see items evicted before their expiration time.
that said, memcached is a cache and should be treated as such unless
you jump through hoops to make it more deterministic (e.g. what i
described in my most recent mail to the list)
--
awl
Martin Grotzke
Martin Grotzke
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
iD4DBQFLnJYsfZ+RkG8quy0RAqt8AJoCTvx1wPJE6Q4P7+rz8Pvi2l2HLgCYvhpa
SBop1pFUnyf6ODozse9kyA==
=c9w0
-----END PGP SIGNATURE-----
moses wejuli
unless you have abysmally low/poor server specs, i think you shouldn't worry about perfomance issues with regard to DB-based session handling -- enhanced with memcached!
Les Mikesell
>
> Tomcat sounds like such a pisser :P Even with your backup thing I'd
> probably still add an option to allow it to journal to a database, and I
> say this knowing how to get every last ounce of efficiency out of
> memcached.
>
> Tomcat provides a PersistentManager ([1]) which allows to store sessions
> in the database. But this manager backups all sessions in batches every
> 10 seconds. For one thing scalability of the application is then
> directly dependent from the database (more than it is already if a
> database is used) and there's a timeframe where sessions can be lost. If
> the session backup frequency is shortened, the database is hit more
> often. Additionally sessions are stored in the database again and again
> even if they were not changed at all. That was the reason why I decided
> not to use this.
What about tomcat's ClusterManager? Doesn't that provide replication across
server instances?
--
Les Mikesell
lesmi...@gmail.com
Martin Grotzke
What about tomcat's ClusterManager? Doesn't that provide replication across server instances?
Adam Lee
the opinion that traditional sessions should be avoided if you want to
truly scale.
On Sunday, March 14, 2010, Martin Grotzke <martin....@googlemail.com> wrote:
> On Sun, Mar 14, 2010 at 5:37 PM, Les Mikesell <lesmi...@gmail.com> wrote:
>
> What about tomcat's ClusterManager? Doesn't that provide replication across server instances?Yes, but the DeltaManager does an all-to-all replication which limits scalability. The BackupManager does a replication to another tomcat (according to the docs this is not that much tested), and this requires special configuration in the load balancer to support this.
>
> And both are using java serialization - for the memcached-session-manager I implemented xml based serialization that allows easier code upgrades. Of course you still need to think about code changes that affect classes stored in the session, but removing fields is easier to support than with java serialization.
>
> Cheers,Martin
>
> --
> Les Mikesell
> lesmi...@gmail.com
>
>
>
> --
> Martin Grotzke
> http://www.javakaffee.de/blog/
>
--
awl
Les Mikesell
> well, it depends on what you mean by scalability... i'm personally of
> the opinion that traditional sessions should be avoided if you want to
> truly scale.
And yet, everyone wants dynamic pages custom-generated to the user's
preferences. So how do you reconcile that? You can help things a bit by
splitting pages into iframe/image components that do/don't need sessions, and
you can make the client do more of the work by sending back values in cookies
instead of just the session key, but I'm not sure how far you can go.
--
Les Mikesell
lesmi...@gmail.com
Martin Grotzke
there seems to be a general phobia out there of storing sessions in the DB
The memcached-session-manager also serves sessions from local memory for optimal performance, so that for a single request there's no I/O needed. Writing sessions to memcached after the request is finished can be done asynchronously. And if such a write would fail occasionally the application can still work correctly as the up-to-date session is still in local memory (the primary data source).
Martin Grotzke
Adam Lee
awl
Martin Grotzke
--
awl
Adam Lee
information is contained in the request-- if you want to pass state
information to the next request, it's easiest to encode it in a sort
of cryptographically secure "token." I find it easiest to think of it
as almost like an FSM where the cookies, token and query parameters
are inputs. Obviously it's not purely deterministic or referentially
transparent, since things do end up getting written to the datastores
and such, but it is a somewhat useful abstraction.
--
awl
dormando
> memcached (with the session timeout as expiration, e.g. 3600). It's still held in the local session map, memcached is just there as a kind of backup device. The session is only pulled from memcached, if a
> tomcat originally serving this session (say tomcat1) died and therefore another tomcat (tomcat2) is asked to serve this session. tomcat2 then does not have a session for the requested sessionId and
> therefore loads this session from memcached.
>
> The case that I was describing before (sending only modified sessions to memcached) is a feature for performance optimization: the asumption is, that there are more requests just accessing the session but
> less requests that actually modify the session. So the idea is that I don't have to update a session in memcached that was accessed but not modified. The issue that had to be handled then was only the case
> of the different timeouts/expiration times:
> - a session was first created/updated and stored in memcached: both in tomcat and memcached it has an expiration of 1h
> - this session is accessed 10 minutes later; as it was not modified it is not updated in memcached. Then this session has an expiration of 1 hour again in tomcat, but in memcached it's already 10 minutes
> old, so it would expire 50 minutes later.
>
> To prevent this premature expiration I used the mentioned background thread to update the expiration of such items in memcached to the remaining expiration time they have in tomcat. In the example above
> nearly 50 minutes later the session would be updated with an expiration time of 10 minutes.
For sake of argument I think that background thread should just journal to
DB, but eh :P
I had a thought; You could use ADD to "ping" the sessions every time
they're accessed. When a session is served from local memory, do an async
ADD of a 0 byte value with a 1 second expiration time against that
session. If the add command fails it actually promotes the existing item
to the head of the LRU. If it doesn't fail you probably want to SET the
session back into memcached.
Feels like your background thread is a little wrong as well. If you can
journal in the session somewhere the last time it was written to
memcached, your thread could update purely based on that. ie; if they're
off by 5+ minutes, sync them asap. So then:
- session created, 1hr expiration. session notes it is "clean" with
memcached as of that second.
- session accessed 10 minutes later, not modified. 1hr expiration, 50 mins
in memcached. Session is pinged via ADD and moves to head of LRU. Fresh.
- 5 minutes later, background thread trawls through sessions in local
memory whose "memcached clean" timestamp is 5+ minutes off from its last
accessed time. syncs to memcached, updates session locally to note when it
was synced? Session is still relatively "fresh" (last accessed 10 minutes
ago). Bumping it to the top of the LRU isn't as bad of a problem as
bumping a 50 minute old session to the top for no reason.
- 55 minutes later, session expires from tomcat. Thread issues DELETE
against memcached which cleans up the session, if it's still there.
> That's already done: sessions that are expiring in tomcat are deleted from memcached
>
> The issue that I described regarding sessions, that were only accessed by the application but not modified and therefore were not updated in memached was the following: when such a session (session A) is
> updated in memcached (just before it would expire in memcached) with a new expiration time of say then 10 minutes (the time that it has left in tomcat), it will be pushed to the head of the LRU.
> Another session (session B) might have been modified just 20 minutes before and sent to memcached with an expiration of 60 minutes, this one will be closer to the tail of the LRU than session A, even if
> session B will still have 40 minutes to live - 30 minutes more than session A. And because it's closer to the tail session B might be dropped before session A, even if session A would already be expired.
>
> However, this would only be an issue if there are too many sessions for the available memory of a slab.
Yeah. Think what I described above solves almost everything :p granted you
can add that sync timestamp. That's essentially the same algorithm I
describe for syncing to a database in my old post, + the ADD trick. Guess
I should go rewrite the post.
This sorta gates on your background thread being able to update a
timestamp in the local memory session though... Failing that you still
have some workaround options but I can't think of any non-ugly ones.
I'll push again on my main point though... Don't overengineer the
slabbing unless you prove that it's a problem first. 1.4 series have many
counters available to tell if one slab class does not have enough memory
relative to other slabs. If you can prove that happens often, you'll need
to check this out more carefully.
> Tomcat provides a PersistentManager ([1]) which allows to store sessions in the database. But this manager�backups all sessions in batches every 10 seconds. For one thing scalability of the application is
> then directly dependent from the database (more than it is already if a database is used) and there's a timeframe where sessions can be lost. If the session backup frequency is shortened, the database is
> hit more often. Additionally sessions are stored in the database again and again even if they were not changed at all. That was the reason why I decided not to use this.
Yeah, I get why the original one is crap. I just don't see why the swing
from "it batches everything every 10 seconds because some guy lacked total
clue" to "holy _SHIT_ we can't touch the database ever!!!" is necessary.
In my old post I basically describe something where:
- A new session gets INSERT'ed into DB, and memcached.
- Fetched from memcached and updated back into memcached.
- On fetch, if you haven't synced with the DB in more than N minutes, sync
with the DB again.
- Alternatively a background thread could trawl through "select
session_id, sync_timestamp, last_known_fetch" from session where etc"
every few minutes and sync stuff back to the DB if it's been changed (but
it doesn't have to keep checking if the last_accessed time isn't being
updated with the synced time).
- At some point background process or whatever DELETE's expired crap from
the DB.
So you still do some writes, but it should be vastly reduced compared to
updating the "last accessed" timestamp on every view, and reads against
the DB should almost never happen for active sessions that stay in cache.
Given the addition of the ADD bit, I guess you could build it so that in a
pinch you could just shut off DB journaling to deal with some overload
scenario.
-Dormando
Ren
On Mar 15, 5:57 pm, Adam Lee <a...@fotolog.biz> wrote:
Yes, I like the implementation outlined in the "A secure cookie
protocol" paper, http://scholar.google.co.uk/scholar?cluster=1658505294292503872&hl=en&as_sdt=2000
And using technologies like ESI (http://en.wikipedia.org/wiki/
Edge_Side_Includes ) where you can have HTML fragments which return
vary heads on the cookies, can move the session problem from the
application server to a HTTP accelerator (Varnish/Squid3/.. ) in front
of it.
Jared
Martin Grotzke
> Yes. As described in my previous post, all necessary state
> information is contained in the request-- if you want to pass state
> information to the next request, it's easiest to encode it in a sort
> of cryptographically secure "token." I find it easiest to think of it
> as almost like an FSM where the cookies, token and query parameters
> are inputs. Obviously it's not purely deterministic or referentially
> transparent, since things do end up getting written to the datastores
> and such, but it is a somewhat useful abstraction.
implemented like this I can have a look at?
Cheers,
Martin
Martin Grotzke
> I had a thought; You could use ADD to "ping" the sessions every time
> they're accessed. When a session is served from local memory, do an async
> ADD of a 0 byte value with a 1 second expiration time against that
> session. If the add command fails it actually promotes the existing item
> to the head of the LRU. If it doesn't fail you probably want to SET the
> session back into memcached.
> Feels like your background thread is a little wrong as well. If you can
> journal in the session somewhere the last time it was written to
> memcached, your thread could update purely based on that. ie; if they're
> off by 5+ minutes, sync them asap. So then:
>
> - session created, 1hr expiration. session notes it is "clean" with
> memcached as of that second.
>
> - session accessed 10 minutes later, not modified. 1hr expiration, 50 mins
> in memcached. Session is pinged via ADD and moves to head of LRU. Fresh.
>
> - 5 minutes later, background thread trawls through sessions in local
> memory whose "memcached clean" timestamp is 5+ minutes off from its last
> accessed time. syncs to memcached, updates session locally to note when it
> was synced? Session is still relatively "fresh" (last accessed 10 minutes
> ago). Bumping it to the top of the LRU isn't as bad of a problem as
> bumping a 50 minute old session to the top for no reason.
>
> - 55 minutes later, session expires from tomcat. Thread issues DELETE
> against memcached which cleans up the session, if it's still there.
Sounds like a good solution for the potential issue, and adding the
"clean" timestamp
is no issue.
Thanx && cheers,
Martin
Adam Lee
<martin....@googlemail.com> wrote:
> Ok, thanx for sharing your experience. Do you have some app online
> implemented like this I can have a look at?
--
awl
Martin Grotzke
Cheers,
Martin
> To unsubscribe from this group, send email to memcached+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.