Memcached make many SI (Software Interrupts)

[ARccRA]

Hi

We have an 2 x quad core server with 32 gb ram. If many clients
connect to this server (only memcached runs on it) the first core run
to nearly 100 % use by si (software interrups) and so some client
can't reach the server.
Memcached runs currently with 4 threads and with version (1.4.2). All
other cores have 70 % idle so I ask me is there a possibility to
improve the performance?

[dormando]

This is an issue with how your network interrupts are being routed, not
with how memcached is being threaded.

Wish I had some good links offhand for this, because it's a little obscure
to deal with. In short; you'll want to balance your network interrupts
across cores. Google for blog posts about smp_affinity for network cards
and irqbalance (which poorly tries to automatically do this).

Depending on how many NIC's you have and if it's multiqueued or not you'll
have to tune it differently. Linux 2.6.35 has some features for extending
the speed of single-queued NIC's (find the pages discussing it on
kernelnewbies.org).

[Jay Paroline]

We've run into this exact same issue and narrowed it down to the NIC,
but don't really know where to go from there. I'm going to look up
Dormando's suggestions but if anyone else has experience with this and
can point us in the right direction, it would be greatly appreciated.

Thanks,

Jay

[Rohit Karlupia]

This is a bit complicated and I still don't understand it fully, but here are few pointers. BTW all that follows has nothing specific to memcached.

1) cat /proc/interrupts

    This will show the list of which core(s) are handling the interrupts for given devices.

    Check eth0/eth1 whichever you are using. The first column is the irq id.

    

2) cat /proc/irq/<irq-number>/smp_affinity 

    If things are good, you should be able to do "echo "xxxx" >  /proc/irq/<irq-number>/smp_affinity to modify the smb affinity. You can do a bit of googling to find about this.

If this doesn't works, it is either your NIC, or its driver or the kernel.

For our hardware centOS (2.6.18) was not spreading the interrupts but with Ubuntu (2.6.32) this worked. 

I have seen this working with kernel 2.6.24 with Intel NICs.

If nothing else works try setting up multiple memcached instances on the same machine using different NICs. For example if you have 4 NICs, create 4 memcached instances each with 7/8GB memory. Usually OS will assign different NICs to different cores (check with cat /proc/interrupts). So let memcached1 listen on eth0, memcached2 on eth1...etc. 

Hope this helps.

thanks!

rohitk

[dormando]

It's a big topic, here's the stuff to google:

- cat /proc/interrupts - look for "eth0", "eth1", etc. If you have one
interrupt assigned to eth0, you have a single-queue NIC.

If you have many interrupts that look like "eth0-0, eth0-1", etc, you have
a multi-queue NIC. These can have their interrupts spread out more.

Use either irqbalance (probably a bad idea) or echoing values into
/proc/irq/nn/smp_affinity (google for help with this), to spread out the
interrupts. You can then experiment with using `taskset` to bind memcached
to the same CPUs as the interrupts, or to different CPU's and see if the
throughput changes.

- look up "linux sysctl network tuning"

This tends to give you crap like this:
net.ipv4.ip_local_port_range = 9500 65536
net.core.rmem_max = 1048576
net.core.wmem_max = 1048576
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 43690 4194304
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_syn_backlog = 16384
#net.ipv4.tcp_synack_retries = 2
net.core.netdev_budget=1000
net.ipv4.tcp_max_tw_buckets = 1512000

Which will vary by if you're using persistent connections or not (ie; how
high the turnover is). Don't blindly copy/paste this stuff.

- Read up on all the "ethtool" options available for your NIC. ensure the
defaults work.

NICs are all configured for a balance between packet latency and
throughput. The more interrupts they coalesce within the driver, the
higher potential latency of packet return. This can be really hard to go
through and the settings will vary by every NIC you fiddle with. I've
managed to get differences of 40k-60k pps by tuning these values. Often
the latency doesn't get much worse.

- Use a recent kernel.

On a particular piece of recent hardware I doubled packet throughput by
moving from 2.6.27 to 2.6.32. I was not able to push 2.6.18 hard without
having it drop networking.

- Use a more recent kernel

http://kernelnewbies.org/Linux_2_6_35#head-94daf753b96280181e79a71ca4bb7f7a423e302a

I haven't played with this much yet, but it looks like a BFD, especially
if you're stuck with single-queue NIC's.

- Get a better NIC.

10ge NICs have awesomesauce features for shoveling more packets around.
There're different levels of how awesome straight-gbit NICs are as well. I
like the high-end intels more than most of the broadcoms, for instance.

- Don't think running multiple instances of memcached will make much of a
difference.

Maybe run more threads though, or try pinning them to a set of CPU's or a
particular CPU.