Best Architecture For DNS Round Robin + Memcached

[Robert Hanson]

Dear all,

I have a plan to start a new project with this scenario :

1 server for database
1 server for web
1 server for memcache

In the future, if the traffic increase, I will add more servers as web
server and also memcache. But many developers suggest me to use this
scenario :

1 server for database
1 server for web + memcache
1 server for web + memcache

Of course with this second scenario, everytime I buy a new server will
be configured as web + memcache server.

I don't mind at all, but the question is...

If i'm using DNS round robin as load balancing method, then user A is
handled by server 1, his PHP session also stored on that server 1, but
on the second request he is handled by server 2, he won't get his
session back, right?

How to make this memcache server can 'comunicate' each other? So that
user's session won't be broken.

Thanks.

[Henrik Schröder]

On Thu, Feb 2, 2012 at 10:16, Robert Hanson <m...@roberthanson.net> wrote:

I don't mind at all, but the question is...

If i'm using DNS round robin as load balancing method, then user A is
handled by server 1,

Yes.

 

his PHP session also stored on that server 1,

No. It will be stored on whatever memcached server in your cluster that is responsible for the specific key that session data is stored under. The whole point of memcached is that it is a distributed cache. Each client (your PHP web app in this case) knows of all memcached servers in the cluster, and knows which keys are stored on which server. That way, every object you store only gets stored on one of the servers, and everyone knows how to get it.


/Henrik

[Robert Hanson]

Hi Henrik, thanks for the answer. You just clear my mind.

[Yiftach Shoolman]

You just have to configure all your Memcache severs (IP list) with
every Memcache client, and it's done.

That being said, I would suggest that you take the first direction
(I.e. deploying Memcache on a dedicated node), this way you decouple
your Memcache scaling from Web scaling.

Also u should take into account that in many cases Memcache generates
a lot of network I/O which can slow down your web/app performance if
deployed on the same node.

Sent from my iPhone

[Henrik Schröder]

On Thu, Feb 2, 2012 at 14:05, Yiftach Shoolman <yiftach....@gmail.com> wrote:

That being said, I would suggest that you take the first direction
(I.e. deploying Memcache on a dedicated node), this way you decouple
your Memcache scaling from Web scaling.

If you have few servers, then this means you will have a few webservers with a lot of unused RAM, and a few memcached servers with a lot of unused CPU. That's a lot of unused hardware, which is rather silly if you're on a tight budget.

Also, both the amount of webservers you need, and the amount of cache memory you need is usually proportional to your web traffic, so it makes little sense to do this decoupling.

As always, benchmark and monitor your app to find out what's best for you. If you have lots of spare RAM on your websevers, put memcached on them, If you don't, don't do it. If you need more webservers than cache servers, simply don't put memcached on all of them. Or do it anyway. If you need more cache servers than webservers, don't put your webapp on the extra ones. Or do it anyway. But as a *general* rule, putting memcached on your webserver machines is a good idea.

 

Also u should take into account that in many cases Memcache generates
a lot of network I/O which can slow down your web/app performance if
deployed on the same node.

This makes absolutely no sense at all. If you have two servers and put a webserver and memcached on both, then *half* of all cache requests will be served by localhost and cause no network I/O, and half will go over the network. But if you put a webserver on one and memcached on the other, then *all* cache requests will be over the network, thus doubling the amount of network I/O for the same amount of web requests.

/Henrik

[Yiftach Shoolman]

TIf you but the Memcached on a dedicated server each webserver only deals with the network I/O associated with its traffic, leaving the dedicated Memcached server to deal with all cached traffic.

To clear it more, if you have N servers each deployed with a webservers a memcached server, and memcached is distributed across all servers,   each webservers needs to deal with Memcached network I/O associated with N-1 webservers --> we found it architecturally wrong, it actually slows down the entire application  

/Henrik

--
Yiftach Shoolman
+972-54-7634621

[Les Mikesell]

On Fri, Feb 10, 2012 at 2:21 AM, Yiftach Shoolman
<yiftach....@gmail.com> wrote:
>
>> This makes absolutely no sense at all. If you have two servers and put a
>> webserver and memcached on both, then *half* of all cache requests will be
>> served by localhost and cause no network I/O, and half will go over the
>> network. But if you put a webserver on one and memcached on the other, then
>> *all* cache requests will be over the network, thus doubling the amount of
>> network I/O for the same amount of web requests.
>
>
> TIf you but the Memcached on a dedicated server each webserver only deals
> with the network I/O associated with its traffic, leaving the dedicated
> Memcached server to deal with all cached traffic.

Normally you would have separate front/back end networks for web and
internal traffic especially if you are scaling to a point where
either might be saturated. There should be little conflict.

> To clear it more, if you have N servers each deployed with a webservers a
> memcached server, and memcached is distributed across all servers,   each
> webservers needs to deal with Memcached network I/O associated with N-1
> webservers --> we found it architecturally wrong, it actually slows down the
> entire application

If you have one memcached server and it goes down, you lose 100% of
your caching.

--
Les Mikesell
lesmi...@gmail.com

[Henrik Schröder]

On Fri, Feb 10, 2012 at 09:21, Yiftach Shoolman <yiftach....@gmail.com> wrote:

TIf you but the Memcached on a dedicated server each webserver only deals with the network I/O associated with its traffic, leaving the dedicated Memcached server to deal with all cached traffic.

You are still not making any sense. Let's say you have two machines, that you need to handle 1000 web requests, and that each web request results in a memcached request.

If you split it like you suggest and put a webserver on one machine, and memcached on the other machine, then the webserver will need to handle 1000 web requests, and the memcached server will need to handle 1000 memcached requests.

But if you don't split it like I suggest, and you put a webserver on each machine and memcached on each machine, then each webserver will need to handle 500 webrequests, but only 250 of those need to talk to memcached over the network, the other half talk to the local memcached and generate no network traffic.

So if you don't split them, there will be 500 memcached requests less over the network, which means you scale better. Not to mention the fact that you're using RAM you probably wouldn't use otherwise, and that you lose less of your cache if one server goes down.
 

To clear it more, if you have N servers each deployed with a webservers a memcached server, and memcached is distributed across all servers,   each webservers needs to deal with Memcached network I/O associated with N-1 webservers --> we found it architecturally wrong, it actually slows down the entire application  

So what? The number of servers you send requests to matters very little, it's the total amount of requests that's interesting. One webserver sending 1000 requests to one memcached server is the same as one webserver sending 100 requests to 10 different servers. It's still 1000 requests?


/Henrik

[Dieter Schmidt]

The point is your example. 1000 req can cause in practice 10000 memcache req. So if you want scale 5 times not only the number of ports on your (one/one) machine setup will hit a hard limit.

Both services on the same machine is also difficult.pppppppppppp

Henrik Schröder <skr...@gmail.com> schrieb:

[Yiftach Shoolman]

Henrik,

I tend to agree with you on the network I/O stuff, but only in case the webserver and the memcached are deployed on All nodes.

In all the other cases I totally don't agree with you. When webserver and memcached are on the same node, you have scaling dependencies, i.e. if you want to scale the webserver, u must scale also scale the memcached unless u insist to have a strange configuration where only part of the your nodes have both webserver and memcached. This creates unbalanced web-tier configuration, and in this case these nodes with both webserver and memcached are also becoming network I/O bound. 

This also goes with the opposite direction, you don't want to add more webservers when all your need is memory to your memcached.

Last but not least is what Dieter said, when the memcached load increases it can easily get to the point where it reduces your webserver performance. This is of course against the whole concept of memcached, which intends to improve performance.

Yiftach

--
Yiftach Shoolman
+972-54-7634621