CPU load, because service is hanging

[Eirik Zakariassen]

Hi again Marc.

I've noticed something interesting lately where CPU load spikes on the nodes running tacplus, it seems like when the service spawns additional workers, they just hang in a ~50-100% load state and never die properly, main service still runs, but machine gets sluggish and I get warnings in vcenter.

If I kill those hanging services it is fine.

Seen anything like that before ?

top - 10:31:13 up 19:08,  2 users,  load average: 5.87, 5.47, 5.23
Tasks: 333 total,   7 running, 326 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17.3 us, 82.1 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.2 si,  0.0 st
MiB Mem :  15732.3 total,   8310.1 free,   1976.6 used,   6090.2 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  13755.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
    977 root      20   0   21276  11404   7432 R  83.1   0.1 901:05.61 tac_plu+
   1954 root      20   0   23640  11472   7740 R  77.4   0.1 900:55.76 tac_plu+
   1990 root      20   0   21416  11180   7744 R  77.1   0.1 901:16.74 tac_plu+
   2062 root      20   0   21276  11712   7768 R  76.7   0.1 899:44.29 tac_plu+
   1981 root      20   0   23640  11436   7736 R  75.4   0.1 901:18.51 tac_plu+
   1072 root      20   0  680520  56924  54672 S   1.7   0.4  10:55.39 rsyslogd
  60427 root      20   0   21272  11340   7912 S   1.0   0.1   0:00.56 tac_plu+
    730 root      20   0  137672  65164  63668 S   0.3   0.4   2:18.17 systemd+
  60177 root      20   0       0      0      0 I   0.3   0.0   0:00.41 kworker+
      1 root      20   0  174744  17416  11252 S   0.0   0.1   0:54.41 systemd  
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.02 kthreadd 


● tac_plus.service - TACACS+ Service
     Loaded: loaded (/etc/systemd/system/tac_plus.service; enabled; preset: dis>
     Active: active (running) since Wed 2026-01-14 15:26:48 CET; 19h ago
   Main PID: 2191 (tac_plus-ng)
      Tasks: 6 (limit: 100406)
     Memory: 839.9M (peak: 2.2G)
        CPU: 3d 3h 12min 46.061s
     CGroup: /system.slice/tac_plus.service
             ├─ 2191 "tac_plus-ng: 0 connections, accepting up to 300 more"
             ├─60427 "tac_plus-ng: 0 connections"
             ├─60428 perl /usr/local/lib/mavis/mavis_tacplus_ldap.pl
             ├─60429 perl /usr/local/lib/mavis/mavis_tacplus_ldap.pl
             ├─60430 perl /usr/local/lib/mavis/mavis_tacplus_ldap.pl
             └─60431 perl /usr/local/lib/mavis/mavis_tacplus_ldap.pl

Jan 15 10:30:12 hostname.domain tac_plus-ng[59831]: 10.0.0.0 >
Jan 15 10:30:12 hostname.domain tac_plus-ng[59831]: 10.0.0.0 >
Jan 15 10:30:12 hostname.domain tac_plus-ng[59831]: 10.0.0.0 >
Jan 15 10:30:12 hostname.domain tac_plus-ng[59831]: 10.0.0.0 >
Jan 15 10:30:16 hostname.domain tac_plus-ng[59831]: - Exiting.
Jan 15 10:30:16 hostname.domain tac_plus-ng[60423]: - Exiting.
Jan 15 10:30:16 hostname.domain tac_plus-ng[2191]: scm_send_msg: s>
Jan 15 10:30:16 hostname.domain tac_plus-ng[2191]: scm_send_msg (/>

[Marc Huber]

Hi Eirik,

that obviously shouldn't happen. Could you please share your "tac_plus-ng -v" output?

Thanks,

Marc

--
You received this message because you are subscribed to the Google Groups "Event-Driven Servers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to event-driven-ser...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/event-driven-servers/2d908953-be09-4579-9f0a-bb40e3844b74n%40googlegroups.com.

[Eirik Zakariassen]

Hi Marc.

This is the output

/usr/local/sbin/tac_plus-ng -v
tac_plus-ng version 7bc67dc7f721abd73106826d89c96f49acd63aea/PCRE2/ARES 1.19.1/CURL 7.76.1/OpenSSL 3.5.1 1 Jul 2025

[Marc Huber]

Hi Eirik,

that GIT hash time stamp is Sun Nov 2 10:07:40 2025 +0100 -- please update to the current GIT, I'm pretty sure that that that the issue you're seeing is already fixed.

In case that busy loop still persists: you can uncomment the "#define DEBUG_CB_STATS" in misc/io_sched.h and rebuild, that enables debug stats which might give some insight on the callbacks involved.

Cheers,

Marc

To view this discussion visit https://groups.google.com/d/msgid/event-driven-servers/5f567eac-37f5-42e7-9ddf-e4f26e3f5482n%40googlegroups.com.

[Eirik Zakariassen]

Thanks Marc.

Did an upgrade today

tac_plus-ng version ab4b5d2594e21984aa35a30bfc7a069cd430c2b4/PCRE2/ARES 1.19.1/CURL 7.76.1/OpenSSL 3.5.1 1 Jul 2025

I'll keep an eye on it.