Question regarding possibly LAN card bottlenecks when using memcached

[Scaler]

Hello,

In production environment I have seen CPU usage go up in PHP servers, with PHP profiler showing possibly memcached(or DB or something else) as culprit. Profiler shows definite memcached being slow but also other things too, so not definite.

memcached server hardware(and hardware connecting to it) is

Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

with 10GBASE-T, short less than 5 m cat 7 cable hooked up to Cisco Nexus switch as layer 2 network.

I noticed at about 400,000 memcached requests per second and at about 3~5 Gbps traffic the above problems occur(many small requests per second).

Benchmarking the LAN card with simply iperf iperf3 I can easily get 9.x Gbps flow.

Do you think the LAN card could be the limiting factor is such case(many small packets)?

Or should such LAN cards be sufficient for such traffic and I should look elsewhere for the performance issues.

Thank you.

[Franklin Ferreira de Lima]

O suporte varia de acordo com as capacidades técnicas do sistema operacional e do ciclo de vida do suporte.

Franklin Ferreira de Lima

--

---
You received this message because you are subscribed to the Google Groups "memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/c2c68e33-3683-46e3-8f42-5b7f0040c768n%40googlegroups.com.

[dormando]

Hey,

Definitely not enough information from what you provide here. Typically
"CPU usage goes up" isn't correlated to "memcached is slow". You usually
see CPU usage go down, because the servers are waiting on data over the
network and are thus idle.

This can change in a few ways, ie; if your PHP servers have low timeouts
and are spin-looping/etc. Or some change in traffic is making lots of
calls to memcached, which translates to lots of syscalls, which translates
to CPU usage on the PHP servers.

Sorry, unfortunately this is too general of an issue. It's common to
wonder if an issue is memcached or not, and we have this wiki page to help
troubleshoot that: https://github.com/memcached/memcached/wiki/Timeouts -
don't skip steps! use the conn tester, read it carefully, etc.

[Scaler]

Thank you for your reply.

In the below case, is the memcached hardware(CPU/LAN card) the limit?

Here is more detailed debugging:

We noticed at over 300,000 memcached requests per second and at about 2.5~3 Gbps traffic memcached suspected problems occur.

memcached server hardware(and hardware connecting to it) is

Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

with 10GBASE-T, short less than 5 m cat 7 cable hooked up to Cisco Nexus switch as layer 2 network.

CPU: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz

OS: Redhat Enteprise Linux 8.5

memcached config:

MAXCONN="600000"
CACHESIZE="30000"
OPTIONS="-l {internalip_redacted} -t 8"

memcached version:

memcached-1.6.14-1.el8.remi.x86_64

debug information:

Regarding: https://github.com/memcached/memcached/wiki/Timeouts

listen_disabled_num -> normal

Is the machine in swap? no

Is the machine overloaded? it seems so in below graph but these are servers dedicated fully to memcached

Unfortunately could not run mc_conn_tester.pl when problems occurred. Also timeouts were not noted, only memcached requests and connections slowing down.

Memcached CPU graph %:

Memcached CPU graph detailed:

Memcached requests per second:

App profiler showing although other portions slowing down, mostly memcached is slowing down severely:

Thank you.