Question regarding possibly LAN card bottlenecks when using memcached

71 views
Skip to first unread message

Scaler

unread,
Apr 11, 2022, 12:47:33 PMApr 11
to memcached
Hello,

In production environment I have seen CPU usage go up in PHP servers, with PHP profiler showing possibly memcached(or DB or something else) as culprit. Profiler shows definite memcached being slow but also other things too, so not definite.

memcached server hardware(and hardware connecting to it) is
Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
with 10GBASE-T, short less than 5 m cat 7 cable hooked up to Cisco Nexus switch as layer 2 network.

I noticed at about 400,000 memcached requests per second and at about 3~5 Gbps traffic the above problems occur(many small requests per second).
Benchmarking the LAN card with simply iperf iperf3 I can easily get 9.x Gbps flow.

Do you think the LAN card could be the limiting factor is such case(many small packets)?
Or should such LAN cards be sufficient for such traffic and I should look elsewhere for the performance issues.

Thank you.

Franklin Ferreira de Lima

unread,
Apr 11, 2022, 1:26:41 PMApr 11
to memc...@googlegroups.com, cq.ma...@light.com.br, qualidade...@light.com.br
O suporte varia de acordo com as capacidades técnicas do sistema operacional e do ciclo de vida do suporte.

Franklin Ferreira de Lima

--

---
You received this message because you are subscribed to the Google Groups "memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/c2c68e33-3683-46e3-8f42-5b7f0040c768n%40googlegroups.com.

dormando

unread,
Apr 11, 2022, 5:27:03 PMApr 11
to memcached
Hey,

Definitely not enough information from what you provide here. Typically
"CPU usage goes up" isn't correlated to "memcached is slow". You usually
see CPU usage go down, because the servers are waiting on data over the
network and are thus idle.

This can change in a few ways, ie; if your PHP servers have low timeouts
and are spin-looping/etc. Or some change in traffic is making lots of
calls to memcached, which translates to lots of syscalls, which translates
to CPU usage on the PHP servers.

Sorry, unfortunately this is too general of an issue. It's common to
wonder if an issue is memcached or not, and we have this wiki page to help
troubleshoot that: https://github.com/memcached/memcached/wiki/Timeouts -
don't skip steps! use the conn tester, read it carefully, etc.

Scaler

unread,
May 24, 2022, 9:40:27 AMMay 24
to memcached
Thank you for your reply.

In the below case, is the memcached hardware(CPU/LAN card) the limit?

Here is more detailed debugging:

We noticed at over 300,000 memcached requests per second and at about 2.5~3 Gbps traffic memcached suspected problems occur.
memcached server hardware(and hardware connecting to it) is
Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
with 10GBASE-T, short less than 5 m cat 7 cable hooked up to Cisco Nexus switch as layer 2 network.
CPU: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
OS: Redhat Enteprise Linux 8.5

memcached config:
MAXCONN="600000"
CACHESIZE="30000"
OPTIONS="-l {internalip_redacted} -t 8"

memcached version:
memcached-1.6.14-1.el8.remi.x86_64

debug information:
listen_disabled_num -> normal
Is the machine in swap? no
Is the machine overloaded? it seems so in below graph but these are servers dedicated fully to memcached

Unfortunately could not run mc_conn_tester.pl when problems occurred. Also timeouts were not noted, only memcached requests and connections slowing down.

Memcached CPU graph %:
memcachedcpu1.png

Memcached CPU graph detailed:
memcachedcpu2.png

Memcached requests per second:
memcachedcommandsperday.png

App profiler showing although other portions slowing down, mostly memcached is slowing down severely:
appprofiler.png

Thank you.
Reply all
Reply to author
Forward
0 new messages