Zeek in Security Onion 16.04 repeatedly

[Kevin Branch]

On just one of my SO 16.04 standalone systems, most days Zeek gets killed one or more times by the kernel because it consumes all memory on the 64GB server.  Looking at our resource monitoring, we see its memory consumption is growing at a fairly linear rate until it saturates memory and gets killed.  Some kind of memory leak?  How do I determine what is behind that?

Our Zeek config should be very close to stock.  The only tweaks I have in place should be these being commented out from local.zeek

@load protocols/ssl/validate-certs

@load protocols/http/detect-sqli

and MailConnectionSummary being set to 0 in zeekctl.cfg.

This is the only one of a number of parallel configured SO 16.04 systems that is exhibiting this and it has been happening for months, with soup upgrades and reboots not making any difference.

Maybe there is some legacy junk hiding somewhere in Zeek's config on this system.  Is there some way I could force a clean rebuild and SO reconfig of Zeek without rebuilding SO from scratch or via sosetup?

dmesg accounts for the oom event:

[2043952.262368] Out of memory: Kill process 5440 (zeek) score 333 or sacrifice child

[2043952.264743] Killed process 5440 (zeek) total-vm:21953056kB, anon-rss:21668912kB, file-rss:131900kB, shmem-rss:0kB
[2043953.458086] oom_reaper: reaped process 5440 (zeek), now anon-rss:0kB, file-rss:131072kB, shmem-rss:0kB

reporter.log looks rather boring

{"ts":"2021-01-11T15:23:03.347279Z","level":"Reporter::INFO","message":"BPFConf filename set: /etc/nsm/peter-mtr/bpf-bro.conf (logger)","location":"/opt/bro/share/zeek/securityonion/./bpfconf.zeek, line 81"}
{"ts":"2021-01-11T15:23:05.551543Z","level":"Reporter::INFO","message":"BPFConf filename set: /etc/nsm/peter-mtr/bpf-bro.conf (proxy)","location":"/opt/bro/share/zeek/securityonion/./bpfconf.zeek, line 81"}
{"ts":"2021-01-11T15:23:06.964550Z","level":"Reporter::INFO","message":"BPFConf filename set: /etc/nsm/peter-mtr/bpf-bro.conf (peter-mtr-3)","location":"/opt/bro/share/zeek/securityonion/./bpfconf.zeek, line 81"}
{"ts":"2021-01-11T15:23:06.968675Z","level":"Reporter::INFO","message":"BPFConf filename set: /etc/nsm/peter-mtr/bpf-bro.conf (peter-mtr-2)","location":"/opt/bro/share/zeek/securityonion/./bpfconf.zeek, line 81"}
{"ts":"2021-01-11T15:23:06.970099Z","level":"Reporter::INFO","message":"BPFConf filename set: /etc/nsm/peter-mtr/bpf-bro.conf (peter-mtr-1)","location":"/opt/bro/share/zeek/securityonion/./bpfconf.zeek, line 81"}
{"ts":"2021-01-11T15:23:06.971799Z","level":"Reporter::INFO","message":"BPFConf filename set: /etc/nsm/peter-mtr/bpf-bro.conf (peter-mtr-4)","location":"/opt/bro/share/zeek/securityonion/./bpfconf.zeek, line 81"}
{"ts":"2021-01-11T15:23:38.934382Z","level":"Reporter::ERROR","message":"Zeek was not configured for GeoIP support (lookup_location(SSH::lookup_ip))","location":"/opt/bro/share/zeek/policy/protocols/ssh/geo-data.zeek, line 30"}
{"ts":"2021-01-11T15:23:45.488285Z","level":"Reporter::ERROR","message":"Zeek was not configured for GeoIP support (lookup_location(SSH::lookup_ip))","location":"/opt/bro/share/zeek/policy/protocols/ssh/geo-data.zeek, line 30"}
{"ts":"2021-01-11T15:24:38.888675Z","level":"Reporter::ERROR","message":"Zeek was not configured for GeoIP support (lookup_location(SSH::lookup_ip))","location":"/opt/bro/share/zeek/policy/protocols/ssh/geo-data.zeek, line 30"}
{"ts":"2021-01-11T15:24:45.427961Z","level":"Reporter::ERROR","message":"Zeek was not configured for GeoIP support (lookup_location(SSH::lookup_ip))","location":"/opt/bro/share/zeek/policy/protocols/ssh/geo-data.zeek, line 30"}

Thanks for your insight

Kevin

[Pete]

Kevin,

I've seen the same issue, but on only one of my sensors.  I haven't found a solution yet, but the Zeek team is aware of it.  See https://github.com/zeek/zeek/issues/956 for details

--

Pete