Memcache as session server with high cache miss?

[TheOnly92]

We have been having problems with the memcache server storing sessions
data, the cache miss was about 20%, therefore we added in another pool
of memcache servers, but now the cache miss rate is up to 40%. We
adjusted the order of the memcache server pool, so that every web
server shares the same order, we also increased memcache memory up to
256 MB, however still users are reporting continuous problems. What
could be the cause?

[Carlos Alvarez]

I don't know if I am understanding it right, but I'd say that
memcached is not the right tool to store session data.

What it is likely happening is that you have evitions (not enough
memory to hold all the concurrent sessions) and memcached is dropping
items from memory.

You should not design an application which depends on data present in
the memcached to work as expected.

Carlos.

[Nicholas Tang]

I know nothing about your app, but if you increased the total size of the pool and nothing else in the equation (number of users, number of page views, number of sessions, size of session data, etc.) has increased, the miss rate should have gone down.

2 things:
- as the earlier responder pointed out, memcached is not the ideal place to store session data
- 256MB seems like a really small amount of memory for a memcached server - we have multiple GB of data on each server

If your miss rate really has gone up, then my question is if you're actually using the entire group of memcached servers as one pool - sounds like you might not be, which could result in some of the problems...

Nicholas

[Brian Hawkins]

Interesting and a little off topic, but if you say memcached is not a good place to put session data then what would you suggest?  I've read that memcached is a good solution for storing sessions for multiple web servers.

Brian

[Carlos Alvarez]

On Wed, Mar 10, 2010 at 12:16 AM, Brian Hawkins <bria...@gmail.com> wrote:
> Interesting and a little off topic, but if you say memcached is not a good
> place to put session data then what would you suggest?  I've read that
> memcached is a good solution for storing sessions for multiple web servers.

Well, since I cannot afford losing my sessions data without
compromising user experience (and therefore, compromising my company's
sales, therefore compromising my salary), I'd say that memcached it is
not a good place for it. I'd would recomend a persistent store: a file
system would be my primary choice. A relational database can do the
job, but I think it is too 'overfeatured' to do that.

Regarding the FS, don't be afraid of performance problems: in a unix
box, you'll be reading your files from memory soon without noticing
(this note from PHK explains why:
http://varnish-cache.org/wiki/ArchitectNotes)

Carlos.

[Brian Moon]

It is a fine place to store transient sessions (incoming link,
navigation information, your stuff) as they can be lost. It is a bad
place to store shopping cart sessions or anything that changes the
user's experience because that would really piss off your users/shoppers.

We use memcached for sessions. But, we are not doing ecommerce or
changing the user's experience based on the session data. And when
someone logs in to there "account" to change something or create a new
email alert, we pull the data from the db on every page load.

Brian.
--------
http://brian.moonspot.net/

[Brian Moon]

Um, how do you propose this works across 40 servers?

Brian.
--------
http://brian.moonspot.net/

[Martin Grotzke]

Hi,

I wonder in which case sessions (or cached data more generally) could
be lost before the expiration is reached if there's enough memory left
in memcached. Is this possible, and under which circumstances?

If this should not happen in the normal case (no crash etc) I'd
calculate the required memory for my sessions. E.g. if I expect to
have a maximum of 10000 concurrent sessions per memcached node with an
average session size of 100kb I'd have to run memcached with (at
least) 1gb.

Is there anything wrong with this? What would have to be considered
additionally?

Cheers,
Martin

2010/3/10, Carlos Alvarez <cbal...@gmail.com>:

--
Von meinen Mobilgerät aus gesendet

Martin Grotzke
http://www.javakaffee.de/blog/

[Carlos Alvarez]

On Wed, Mar 10, 2010 at 6:48 AM, Martin Grotzke
<martin....@googlemail.com> wrote:
> Is there anything wrong with this? What would have to be considered
> additionally?

If you have an intranet app, I don't see any problem with this.
If you have a public website, you are exposing yourself to DOS attacks
with really low load.

Carlos.

[Brian Hawkins]

Explain how using memcached opens one self to a DOS attack?

Brian

[Adam Lee]

chances are, the same DOS attack would kill your datastore as well...

i you want speed and persistence, i'd recommend checking out products
like tokyotyrant; it even speaks memcached. we use it to cache user
data and it works incredibly well...

--
awl

[Les Mikesell]

On 3/10/2010 10:28 AM, Brian Hawkins wrote:
> Explain how using memcached opens one self to a DOS attack?

If you are expecting it to be a persistent store by itself, simply
exceeding capacity will drop data before you expect it to expire. Of
course if your sessions are tied to logins this would be harder to
cause, and if you have a backing DB of critical entries then it's not a
problem until you overwhelm the DB.

Memcache itself isn't a problem unless you expect it to always have your
data, which it isn't intended to do.

--
Les Mikesell
lesmi...@gmail.com

[Carlos Alvarez]

On Wed, Mar 10, 2010 at 1:28 PM, Brian Hawkins <bria...@gmail.com> wrote:
> Explain how using memcached opens one self to a DOS attack?

Using memcached to store session data just relying in a previous
calculation of how many memory you will need (all premises) exposes
you to DOS because of the effect explained in other mails: it would be
very easy and some attacker would need a very low bandwith to force
the memcached to expell valid data. If your application relies on data
to be present on the memcached to work properly, you have a problem.

If you have a persistent store, it won't expell data because of space
constraints. Also, I find a lot more easier (and cheaper) to have a
4TB persistent store than a set of memcached.

Carlos.

[Carlos Alvarez]

On Wed, Mar 10, 2010 at 1:52 PM, Carlos Alvarez <cbal...@gmail.com> wrote:
> Also, I find a lot more easier (and cheaper) to have a
> 4TB persistent store than a set of memcached.

Sorry. I should have written: "than a set of memcached totalizing
0.5TB of memory".

Carlos.

[Les Mikesell]

But you'll find it very expensive to scale up the number of servers
accessing that persistent store and the speed it can operate if you
don't use something like memcache in front of it.

--
Les Mikesell
lesmi...@gmail.com

[Carlos Alvarez]

On Wed, Mar 10, 2010 at 2:41 PM, Les Mikesell <lesmi...@gmail.com> wrote:
> But you'll find it very expensive to scale up the number of servers
> accessing that persistent store and the speed it can operate if you don't
> use something like memcache in front of it.

Of course. If It was understood that I was saying that memcached is
useless I apologize for my poor english.

I was trying to point out that if your app. behaves differently
depending whether the data is present on the memcached or not (apart
from response time and scalability), you may be using a cache tool for
other purpouses. And it is risky.

Carlos.

[Marc Bollinger]

> the memcached to expell valid data. If your application relies on data
> to be present on the memcached to work properly, you have a problem.

Now you have two problems.

--
Marc Bollinger
mboll...@gmail.com

[Adam Lee]

Agreed. 

As I said before, we use a multi-layer approach to solve a similar problem.  We use a database as our primary store for user data and it's cached in TokyoTyrant and memcached.  The majority of our page views are read-only and therefore don't need to hit the database.  We pull that data from memcached if it exists and, if not, we pull it from TokyoTyrant and stick it in memcached again.  The DB is never accessed from these pages (and, in fact can't be).

In order to update user data, it is written to all three layers.  This can only be done by logged-in users modifying their own account or by backend jobs that are updating user data-- this means that it happens at a much smaller scale and DOS attacks are much less of a concern (though we do have rate controls in place, obviously).

This allows us to scale to a huge level (several thousand page views a second) that wasn't really previously possible... before we started using TokyoTyrant as the persistent middle-tier, the system used a file-based cache on our SAN that was served over NFS and it didn't scale nearly as well, plus it was a lot harder to administrate, backup, etc...  The same system that was done with millions of files served by 6 very high-end servers (32 cores per machine) is achieved with two boxes running TokyoTyrant in a master-master replication setup. 

--
awl

[moses wejuli]

I agree memcache isn't expected - and designed - to persistently keep your data, hence unsuitable for critical session data storage.... which sweetly introduces an idea for our beloved memcache developers: would it not be nice to have an option (just an option) to cache data with no possibility of expulsion with, for example, a precondition that such cached data cannot exceed n% of total memory allocated to memecache on each server (or quite simply with no preconditions).

This way, large (or small) sets of critical data, e.g. sessions (or a cross section of session data e.g. login data) would be saved in memcache without worrying about expulsions or expirations

My hunch is that the memcache devs have probably considered this idea but never went ahead with it..... I would be interested to know why?

[Dean Harding]

> My hunch is that the memcache devs have probably
> considered this idea but never went ahead with
> it..... I would be interested to know why?

I'm not one of the devs, but I can guess why they wouldn't want to do this.
It's pretty simple: it's not just expiration and expulsion that can cause a
memcache node to lose keys. If you need to reboot a server, the data will be
lost. If you have a hard disk crash and the server dies, you lose your data.

Of course, you can reboot database servers and so on as well, but they're
usually replicated so that there's no interruption of service and no data
lost.

Besides, this feature could affect performance, and memcache is ALL about
performance.

For us, our session data is usually mostly static. When you log on, we
create the session data and it doesn't change much until you log off. So we
actually store it in the database AND in memcache. The database then is
mostly write-only and only needed if a memcache node goes down and the key
is lost.

Dean.

[moses wejuli]

Yes, good point as far as service interruption is concerned but I  don't see how this affects performance, especially if the programmer carefully uses this feature for tiny amounts of data per user.... I just think, given how far memcache has metamorphosed, the time is probably right to include such a feature (to be used with some care of course)

On 10 March 2010 23:04, Dean Harding <dean.h...@dload.com.au> wrote:

> My hunch is that the memcache devs have probably
> considered this idea but never went ahead with
> it..... I would be interested to know why?

[Les Mikesell]

On 3/10/2010 5:14 PM, moses wejuli wrote:
> Yes, good point as far as service interruption is concerned but I don't
> see how this affects performance, especially if the programmer carefully
> uses this feature for tiny amounts of data per user.... I just think,
> given how far memcache has metamorphosed, the time is probably right to
> include such a feature (to be used with some care of course)

You can't count on nothing ever going wrong. How should this
theoretically reliable storage mechanism respond to a network glitch to
one or more of the servers? Or just a miscalculation in terms of how
much data you throw at it?

--
Les Mikesell
lesmi...@gmail.com

[moses wejuli]

it's not so much a "theoretically reliable storage mechanism" but only storing data for as long as the programmer chooses or in this case, for only as long as the user is logged on (data is expelled when the user chooses to log out or is forcibly logged out for one reason or another).

data miscalculation? i think you'd have to take care as a dev for this not to happen -- infact this is presently the case (think max slab size)

[Les Mikesell]

On 3/10/2010 5:46 PM, moses wejuli wrote:
> it's not so much a /"theoretically reliable storage mechanism"/ but only

> storing data for as long as the programmer chooses or in this case, for
> only as long as the user is logged on (data is expelled when the user
> chooses to log out or is forcibly logged out for one reason or another).

Or until any component fails, needs a change, etc.

> data miscalculation? i think you'd have to take care as a /dev/ for this

> not to happen -- infact this is presently the case (think max slab size)

I'd guess that it's uncommon for memcache users to have any idea how
many potential keys might be in use at once (constructing them from
arbitrary sql queries, etc.). Or to do anything to remove data. You
really can't iterate over it to see what needs to be removed - and where
else would you store the keys so you'd know about them?

--
Les Mikesell
lesmi...@gmail.com

[TheOnly92]

We used memcache as a session storage server because we needed to
balance load across servers and share session data. Persistent file
storage is not usable, and since memcache session storage is easy to
configure in PHP, so we have decided to use it.

Before this, memcache maximum storage was set to 64 MB, the data went
up to 54 MB and so we thought it was the cause of cache misses. But
after we increased to 256 MB, the problem is still occurring. Users
report that after they logged in, they click on another page and they
get logged out. But after a refresh they appear logged in again.

session.gc_maxlifetime 1440 1440

[Carlos Alvarez]

On Thu, Mar 11, 2010 at 6:03 AM, TheOnly92 <0504...@gmail.com> wrote:
> We used memcache as a session storage server because we needed to
> balance load across servers and share session data. Persistent file
> storage is not usable, and since memcache session storage is easy to
> configure in PHP, so we have decided to use it.

Why it is not usable?. I've seen (actually I have running) NFS
suporting high loads. Very highs loads, actually.

As others have said, a nosql database would help, too.

> Before this, memcache maximum storage was set to 64 MB, the data went
> up to 54 MB and so we thought it was the cause of cache misses. But
> after we increased to 256 MB, the problem is still occurring. Users
> report that after they logged in, they click on another page and they
> get logged out. But after a refresh they appear logged in again.

Do you have evictions?. If not and the client are not losing the
conection to the memcached and you are not flushing or restarting the
memcached, look for bugs in your appl.

Carlos.

[Jérôme Patt]

TheOnly92 schrieb:

As far as i can see, nobody has asked this questions before: if you want to use memcached
as a session store, how is your setup designed? specialised memcached's (good, good, good)
or one instance per webserver (bad, bad, bad)? how does your php.ini look like (all
entries starting with "session")? Do you have a any or even a large number of evictions on
your memcached's? Is your network saturated (using memcached as a session store produces a
large amount of network traffic, which can lead to connection problems, which could also
explain your issue).

Kind regards,

J�r�me

[Martin Grotzke]

Hi,

I'm trying to follow this thread on my mobile, i hope i didn't miss
anything. But AFAICS it was not yet explained, when memcached might
drop cached data as long as there's enough memory and expiration is
not reached. Or is this not deterministic at all? Perhaps you can
point me to resources providing more details on this?

Cheers,
Martin

2010/3/11, Jérôme Patt <jerom...@googlemail.com>:

> Jérôme

[Ren]

Are you using php pecl memcached, together with a persistent
connection?

Jared

[Les Mikesell]

On 3/11/2010 1:01 PM, Martin Grotzke wrote:
> Hi,
>
> I'm trying to follow this thread on my mobile, i hope i didn't miss
> anything. But AFAICS it was not yet explained, when memcached might
> drop cached data as long as there's enough memory and expiration is
> not reached. Or is this not deterministic at all? Perhaps you can
> point me to resources providing more details on this?

'Enough' memory may not be what you expect unless you understand how
your data fits in the allocated slabs. And I'm not sure what happens if
the keys have hash collisions.

--
Les Mikesell
lesmi...@gmail.com

[Marc Bollinger]

>  And I'm not sure what happens if the keys
> have hash collisions.

Like many key-value stores, last write wins.

- Marc

[Adam Lee]

On Thu, Mar 11, 2010 at 4:03 AM, TheOnly92 <0504...@gmail.com> wrote:

We used memcache as a session storage server because we needed to
balance load across servers and share session data. Persistent file
storage is not usable, and since memcache session storage is easy to
configure in PHP, so we have decided to use it.

Hard to argue with that.  As I said before, though, it might be worth looking into TokyoTyrant.  It speaks memcached and it's persistent, so you can basically drop it in and start using it with your application code as it exists.  It's not as fast or lightweight as memcached, though, so I would suggest using it only as an add-on in situations where persistence is important.

 

Before this, memcache maximum storage was set to 64 MB, the data went
up to 54 MB and so we thought it was the cause of cache misses. But
after we increased to 256 MB, the problem is still occurring. Users
report that after they logged in, they click on another page and they
get logged out. But after a refresh they appear logged in again.

session.gc_maxlifetime  1440    1440

This is definitely unusual behavior.  Have you tried debugging it at all?  I'd say that you should try going through step-by-step and checking the value that's stored in memcached at each point.  Given what you described, it sounds to me like the bug is on the application side, not in memcached-- it sounds like some sort of state is perhaps being stored and that's why they see themselves as logged-out until they do a refresh.  

--
awl

[Jérôme Patt]

TheOnly92 schrieb:

As far as i can see, nobody has asked this questions before: if you want to use memcached

as a session store, how is your setup designed? specialised memcached's (good, good, good)
or one instance per webserver (bad, bad, bad)? how does your php.ini look like (all
entries starting with "session")? Do you have a any or even a large number of evictions on
your memcached's? Is your network saturated (using memcached as a session store produces a
large amount of network traffic, which can lead to connection problems, which could also
explain your issue).

Kind regards,

J�r�me

[elSchrom]

As far as i can see, nobody has asked this questions before: if you

want to use memcached
as a session store, how is your setup designed? specialised
memcached's (good, good, good)
or one instance per webserver (bad, bad, bad)? how does your php.ini
look like (all
entries starting with "session")? Do you have a any or even a large
number of evictions on
your memcached's? Is your network saturated (using memcached as a
session store produces a
large amount of network traffic, which can lead to connection
problems, which could also
explain your issue).

Kind regards,

Jérôme

[Jérôme Patt]

TheOnly92 schrieb:

As far as i can see, nobody has asked this questions before: if you want to use memcached

as a session store, how is your setup designed? specialised memcached's (good, good, good)
or one instance per webserver (bad, bad, bad)? how does your php.ini look like (all
entries starting with "session")? Do you have a any or even a large number of evictions on
your memcached's? Is your network saturated (using memcached as a session store produces a
large amount of network traffic, which can lead to connection problems, which could also
explain your issue).

Kind regards,

J�r�me

[TheOnly92]

Here is our current setup:
webserver1 (also runs session memcache server)
webserver2 (also runs session memcache server)
database (specialized memcache storage for data caching)

We are not really a high loaded site, at peak time only about 1500
users online together. Network is not really saturated, as not much
data being transferred I believe.

Here is the phpinfo() part for sessions:
session.auto_start Off Off
session.bug_compat_42 On On
session.bug_compat_warn On On
session.cache_expire 180 180
session.cache_limiter nocache nocache
session.cookie_domain no value no value
session.cookie_httponly Off Off
session.cookie_lifetime 0 0
session.cookie_path / /
session.cookie_secure Off Off
session.entropy_file no value no value
session.entropy_length 0 0
session.gc_divisor 100 100
session.gc_maxlifetime 1440 1440
session.gc_probability 0 0
session.hash_bits_per_character 4 4
session.hash_function 0 0
session.name PHPSESSID PHPSESSID
session.referer_check no value no value
session.save_handler memcache memcache
session.save_path tcp://172.23.111.11:11211,tcp://172.23.111.12:11211
tcp://172.23.111.11:11211,tcp://172.23.111.12:11211
session.serialize_handler php php
session.use_cookies On On
session.use_only_cookies Off Off
session.use_trans_sid 0 0

And yes, we use PECL memcache extension.

[elSchrom]

Ok, that seems correct for me. Your described issue still sounds for
me like a connection problem.
On which interface is your memcached bound (memcached.conf?)? Does a

telnet 172.23.111.11 11211

work from every webserver instance? I could imagine following
scenario:

User1 opens a session on server1, which is stored in memd3. The next
page impression balances him to
server2, which is not able to connect to memd3 -> His sessions is not
existing for the time of this request.
The next request is again balanced to server1 and the session is
existing again.

[martin.grotzke]

What about setting the minimum slab size to 1 mb (-n 1048576) so that
there's only one slab and one can calculate with this?

Cheers,
Martin

Cheers,
Martin

>
> --
>    Les Mikesell
>     lesmikes...@gmail.com

[Les Mikesell]

On 3/12/2010 11:21 AM, martin.grotzke wrote:

>>
>>> Hi,
>>
>>> I'm trying to follow this thread on my mobile, i hope i didn't miss
>>> anything. But AFAICS it was not yet explained, when memcached might
>>> drop cached data as long as there's enough memory and expiration is
>>> not reached. Or is this not deterministic at all? Perhaps you can
>>> point me to resources providing more details on this?
>>
>> 'Enough' memory may not be what you expect unless you understand how
>> your data fits in the allocated slabs. And I'm not sure what happens if
>> the keys have hash collisions.
> What about setting the minimum slab size to 1 mb (-n 1048576) so that
> there's only one slab and one can calculate with this?

After seeing more of this thread, I'm inclined to think that the problem
that started it is really a misconfiguration or network issue. While
you shouldn't expect memcache to be a reliable store, the miss
percentage should not double when you add another server.

--
Les Mikesell
lesmi...@gmail.com

[martin.grotzke]

I just started a new thread "How to get more predictable caching
behavior - how to store sessions in memcached" to discuss my more
general question on how to achieve more predictability/reliability (I
might have hijacked this thread a little bit, sorry for this).

Cheers,
Martin

>     lesmikes...@gmail.com

[dormando]

Can you paste the output of 'stats' against both of your memcached
servers? Is your configuration identical on both servers? How have you
been calculating the miss rate?

[TheOnly92]

I'm retrieving statistics via Memcache::extendedStats function, here
are the basics:

Session Server 1
Version 1.2.2
Uptime 398,954 sec
Cache Hits 2,065,061
Cache Misses 987,726 (47.83%)
Current Items 381,928
Data Read 4,318,055.02 KB
Data Written 2,011,004.09 KB
Current Storage 100,688.96 KB
Maximum Storage 256.00 MB
Current Connections 9
Total Connections 5,278,414
Session Server 2
Version 1.2.2
Uptime 398,943 sec
Cache Hits 2,225,697
Cache Misses 987,733 (44.38%)
Current Items 381,919
Data Read 4,323,893.05 KB
Data Written 2,159,309.95 KB
Current Storage 100,685.52 KB
Maximum Storage 256.00 MB
Current Connections 11
Total Connections 5,278,282

We are absolutely sure that both webservers are able to access the
memcache server instances, we selected memcache because it was an easy
configuration and setup without any changes of source code required,
not to think that it is "absolutely" reliable. We just need to make
sure that it works most of the time, but current situation is just
unacceptable.

[dormando]

Can you telnet to the instances, type "stats", "stats items", and "stats
slabs", then copy/paste all that into pastebin?

echo "stats" | nc host 11211 >> stats.txt works too

You version is very old... It's missing many statistical counters that
could help us diagnose a problem. The extendedstats isn't printing an
evictions counter, but I can't remember if that version even had one.

Can you describe your problem in more detail? If I recall:

- User logs in.
- Clicks somewhere. now they're logged out?
- They click somewhere else, and they're logged in again? Does this mean
they found their original session again, or did you app log them in again?

-Dormando

[TheOnly92]

Webserver1: http://paste2.org/p/715491
Webserver2: http://paste2.org/p/715492

Situation:
1. User logs in.
2. User clicks somewhere (still logged in)
3. User clicks on another placed and gets redirected to the home page
(appears logged out for this page)
4. Refreshes and able to access the page again (logged in).

[dormando]

From the looks of it, you're not evicting anything. Which means memcached
isn't forgetting about anything you're telling it. For some other reason
your sessions are not being found.

From your description of the problem (sometimes logged in, sometimes not),
along with the high miss rate on memcached, tells me that something else
is wrong with your configuration.

I know you've been asked this before, but:

- Can you quadruple check that both webservers have *identical
configurations*. Both webservers list both memcached servers as the same
IPs, in the same order. If they're incorrect, if a user hits webserver1,
then webserver2, then webserver1, that would end up having them flip
between memcached's.

- Have you combed your error logs? Are you handling errors from your
client? It's possible that memcached or your client are throwing errors
and you're not noticing, or that it's failing to contact one server
sometimes and is then contacting the other.

- Have you tried enabling persistent connections from php? There should be
an option for it. You have a high rate of new connections and switching to
persistent can reduce those symptoms (such as firewalls, running out of
local ports, etc).

- Your software is very old. Memcached is very old and I'm going to guess
that your pecl/memcached library is also very old. You're missing a lot of
counters and bug fixes that would help diagnose (or outright fix) issues
like this. Would you consider upgrading to the latest non-betas?

[Adam Lee]

Given the behavior and the high miss rate, I really have to think that something doesn't match between your two servers.  Either the server configs are different or they're trying to get different keys-- there's no reason, otherwise, that data would appear to be there for one of them but not for the other.

Can you paste your memcached configs from both servers?  It's important to note that even order matters in respect to memcached config.  If the two machines have the same servers but in different orders, they will hash the values to different memcached servers. 

--
awl

[TheOnly92]

Well, before I come consulting you guys, I have asked the same
question at Stackoverflow and someone pointed it out that both servers
need to have the same memcached pool IP order, so we fixed that. Our
pecl/memcache version is 3.0.1, it's not too old? I haven't checked
the error logs yet so I can't tell anything right now, will do it
later on... But how do I check the memcached configs?