memory-leak in cf-execd 3.9.0 on RedHat 6???

[Sven Probst]

Hi,

is it possible that there is a memory-leak in cf-execd in Version 3.9.0 (community-edition, cfengine-community-3.9.0-1.el6.x86_64.rpm ) on RHEL 6?
We see that the daemon is consuming memory over time - this is new since the update to 3.9.0 .
I´m not very familiar with the correct usage of valgrind, but output after 20 min is:

==00:00:19:37.443 30464== HEAP SUMMARY:
==00:00:19:37.443 30464==     in use at exit: 668,148 bytes in 29,147 blocks
==00:00:19:37.443 30464==   total heap usage: 892,784 allocs, 863,637 frees, 177,665,575 bytes allocated
==00:00:19:37.443 30464==
==00:00:19:37.499 30464== 2,310 (32 direct, 2,278 indirect) bytes in 1 blocks are definitely lost in loss record 321 of 385
==00:00:19:37.499 30464==    at 0x4A057BB: calloc (vg_replace_malloc.c:593)
==00:00:19:37.499 30464==    by 0x4C9E4F8: xcalloc (alloc.c:47)
==00:00:19:37.499 30464==    by 0x4CA2CA7: JsonElementCreateContainer.clone.0 (json.c:112)
==00:00:19:37.499 30464==    by 0x4CBC0E6: GetProcFileInfo (unix_iface.c:1060)
==00:00:19:37.499 30464==    by 0x4CBC7ED: GetNetworkingInfo (unix_iface.c:1256)
==00:00:19:37.499 30464==    by 0x4CBB3D8: DetectEnvironment (sysinfo.c:2895)
==00:00:19:37.499 30464==    by 0x4C7D913: GenericAgentDiscoverContext (generic_agent.c:431)
==00:00:19:37.499 30464==    by 0x40439D: main (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==
==00:00:19:37.499 30464== 10,792 (32 direct, 10,760 indirect) bytes in 1 blocks are definitely lost in loss record 370 of 385
==00:00:19:37.499 30464==    at 0x4A057BB: calloc (vg_replace_malloc.c:593)
==00:00:19:37.499 30464==    by 0x4C9E4F8: xcalloc (alloc.c:47)
==00:00:19:37.499 30464==    by 0x4CA2CA7: JsonElementCreateContainer.clone.0 (json.c:112)
==00:00:19:37.499 30464==    by 0x4CBC2EF: GetNetworkingInfo (unix_iface.c:1135)
==00:00:19:37.499 30464==    by 0x4CBB3D8: DetectEnvironment (sysinfo.c:2895)
==00:00:19:37.499 30464==    by 0x4C7D913: GenericAgentDiscoverContext (generic_agent.c:431)
==00:00:19:37.499 30464==    by 0x40439D: main (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==
==00:00:19:37.499 30464== 16,047 (32 direct, 16,015 indirect) bytes in 1 blocks are definitely lost in loss record 372 of 385
==00:00:19:37.499 30464==    at 0x4A057BB: calloc (vg_replace_malloc.c:593)
==00:00:19:37.499 30464==    by 0x4C9E4F8: xcalloc (alloc.c:47)
==00:00:19:37.499 30464==    by 0x4CA2CA7: JsonElementCreateContainer.clone.0 (json.c:112)
==00:00:19:37.499 30464==    by 0x4CBC59B: GetNetworkingInfo (unix_iface.c:1191)
==00:00:19:37.499 30464==    by 0x4CBB3D8: DetectEnvironment (sysinfo.c:2895)
==00:00:19:37.499 30464==    by 0x4C7D913: GenericAgentDiscoverContext (generic_agent.c:431)
==00:00:19:37.499 30464==    by 0x40439D: main (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==
==00:00:19:37.499 30464== 43,890 (608 direct, 43,282 indirect) bytes in 19 blocks are definitely lost in loss record 380 of 385
==00:00:19:37.499 30464==    at 0x4A057BB: calloc (vg_replace_malloc.c:593)
==00:00:19:37.499 30464==    by 0x4C9E4F8: xcalloc (alloc.c:47)
==00:00:19:37.499 30464==    by 0x4CA2CA7: JsonElementCreateContainer.clone.0 (json.c:112)
==00:00:19:37.499 30464==    by 0x4CBC0E6: GetProcFileInfo (unix_iface.c:1060)
==00:00:19:37.499 30464==    by 0x4CBC7ED: GetNetworkingInfo (unix_iface.c:1256)
==00:00:19:37.499 30464==    by 0x4CBB3D8: DetectEnvironment (sysinfo.c:2895)
==00:00:19:37.499 30464==    by 0x404037: StartServer (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==    by 0x404410: main (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==
==00:00:19:37.499 30464== 205,048 (608 direct, 204,440 indirect) bytes in 19 blocks are definitely lost in loss record 384 of 385
==00:00:19:37.499 30464==    at 0x4A057BB: calloc (vg_replace_malloc.c:593)
==00:00:19:37.499 30464==    by 0x4C9E4F8: xcalloc (alloc.c:47)
==00:00:19:37.499 30464==    by 0x4CA2CA7: JsonElementCreateContainer.clone.0 (json.c:112)
==00:00:19:37.499 30464==    by 0x4CBC2EF: GetNetworkingInfo (unix_iface.c:1135)
==00:00:19:37.499 30464==    by 0x4CBB3D8: DetectEnvironment (sysinfo.c:2895)
==00:00:19:37.499 30464==    by 0x404037: StartServer (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==    by 0x404410: main (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==
==00:00:19:37.499 30464== 304,893 (608 direct, 304,285 indirect) bytes in 19 blocks are definitely lost in loss record 385 of 385
==00:00:19:37.499 30464==    at 0x4A057BB: calloc (vg_replace_malloc.c:593)
==00:00:19:37.499 30464==    by 0x4C9E4F8: xcalloc (alloc.c:47)
==00:00:19:37.499 30464==    by 0x4CA2CA7: JsonElementCreateContainer.clone.0 (json.c:112)
==00:00:19:37.499 30464==    by 0x4CBC59B: GetNetworkingInfo (unix_iface.c:1191)
==00:00:19:37.499 30464==    by 0x4CBB3D8: DetectEnvironment (sysinfo.c:2895)
==00:00:19:37.499 30464==    by 0x404037: StartServer (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==    by 0x404410: main (in /var/cfengine/bin/cf-execd)
==00:00:19:37.499 30464==
==00:00:19:37.499 30464== LEAK SUMMARY:
==00:00:19:37.499 30464==    definitely lost: 1,920 bytes in 60 blocks
==00:00:19:37.499 30464==    indirectly lost: 581,060 bytes in 29,040 blocks
==00:00:19:37.499 30464==      possibly lost: 0 bytes in 0 blocks
==00:00:19:37.499 30464==    still reachable: 85,168 bytes in 47 blocks
==00:00:19:37.499 30464==         suppressed: 0 bytes in 0 blocks
==00:00:19:37.499 30464== Reachable blocks (those to which a pointer was found) are not shown.
==00:00:19:37.499 30464== To see them, rerun with: --leak-check=full --show-reachable=yes
==00:00:19:37.499 30464==
==00:00:19:37.499 30464== For counts of detected and suppressed errors, rerun with: -v
==00:00:19:37.499 30464== Use --track-origins=yes to see where uninitialised values come from
==00:00:19:37.499 30464== ERROR SUMMARY: 32558 errors from 141 contexts (suppressed: 6 from 6)

valgrind was called with
# valgrind --time-stamp=yes --log-file=/var/tmp/valgrind-cfexec.%p --leak-check=full /var/cfengine/bin/cf-execd -F

Any help is welcome.

Best regards
  Sven

[Dimitrios Apostolou]

Hi Sven,

I believe this has been fixed in master with commit ed577f1a5bb72c409063949bfadf733b708d2713. I'll make sure it gets cherry-picked for 3.9.1, soon to be released.


Thanks,

Dimitris

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[robin...@gmail.com]

Was the memory leak in cf-execd fix put into community 3.10.0?  Here is the system specifics:

chn-vm-clic01: # cat /etc/redhat-release
CentOS release 6.9 (Final)
chn-vm-clic01: # uname -a
Linux chn-vm-clic01 2.6.32-696.16.1.el6.x86_64 #1 SMP Wed Nov 15 16:51:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
chn-vm-clic01: # cf-agent --version
CFEngine Core 3.10.0
chn-vm-clic01: # cf-execd --version
CFEngine Core 3.10.0

valgrind:

==8480== Memcheck, a memory error detector
==8480== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==8480== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==8480== Command: /var/cfengine/bin/cf-execd --no-fork

==8480== LEAK SUMMARY:
==8480==    definitely lost: 2,672,684 bytes in 787 blocks
==8480==    indirectly lost: 3,233 bytes in 174 blocks
==8480==      possibly lost: 0 bytes in 0 blocks
==8480==    still reachable: 85,236 bytes in 49 blocks
==8480==         suppressed: 0 bytes in 0 blocks
==8480== Reachable blocks (those to which a pointer was found) are not shown.
==8480== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==8480==
==8480== For counts of detected and suppressed errors, rerun with: -v
==8480== Use --track-origins=yes to see where uninitialised values come from
==8480== ERROR SUMMARY: 31439 errors from 156 contexts (suppressed: 4 from 4)

[Nick Anderson]

I believe the specific commit referenced is in 3.10.0.

> $ git tag --contains ed577f1a5bb72c409063949bfadf733b708d2713
3.10.0
3.10.0-build1
3.10.0-build2
3.10.0-build3
3.10.0-build4
3.10.0b1
3.10.0b1-build1
3.10.0b1-build2
3.10.0b1-build3
3.10.1
3.10.1-build1
3.10.2
3.10.2-build1
3.10.2a-PTV
3.10.2a-PTV-build1
3.11.0
3.11.0-build1
3.11.0b1
3.11.0b1-build1
3.11.0b1-build2

I see at least one recent commits that fix a leak in execd.

https://github.com/cfengine/core/commit/076941712c842f6518c207ac8305eb19c4853ff0