Missing info about container activity in UI
Sofia Amfiteatrova
2023 Jul 04 14:59:07 (agent-name) any->/var/log/syslog Jul 4 17:59:06 agent-name dockerd[1224]: time="2023-07-04T17:59:06.472907976+03:00" level=error msg="restartmanger wait error: driver failed programming external connectivity on endpoint app-name (param): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))"
2023 Jul 04 14:59:07 (agent-name) any->/var/log/syslog Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.504134355+03:00" level=warning msg="Failed to allocate and map port 8000-8000: (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))"
2023 Jul 04 14:59:07 (agent-name) any->/var/log/syslog Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.532416050+03:00" level=error msg="cleanup: failed to delete container from containerd: container \"\" in namespace \"\": not found"
2023 Jul 04 14:59:07 (agent-name) any->/var/log/syslog Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.651138519+03:00" level=error msg="restartmanger wait error: driver failed programming external connectivity on endpoint app (): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))"
Luis González Romero
Hello Sofia, hope you’re great.
What’s happening is that timestamp (time="2023-07-04T17:59:06.472907976+03:00") won’t match with the regex from the decoders, you can check it here
So, what you can do is create a sibling decoder within your /var/ossec/etc/decoders/local_decoder.xml, for example:
<decoder name="docker"> <program_name>^dockerd|^containerd</program_name> <prematch>^time="\d+-\d+-\d+T\d+:\d+:\d+.\d+\S+" level=\S+ msg="</prematch> <regex>level=(\S+) msg="(\.+)"</regex> <order>docker.level,docker.message</order> </decoder>Then, the events will be decoded and will trigger an alert:
root@afoansible:/home/vagrant# /var/ossec/bin/wazuh-logtest Starting wazuh-logtest v4.4.4 Type one log per line Jul 4 17:59:06 agent-name dockerd[1224]: time="2023-07-04T17:59:06.472907976+03:00" level=error msg="restartmanger wait error: driver failed programming external connectivity on endpoint app-name (param): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))" **Phase 1: Completed pre-decoding. full event: 'Jul 4 17:59:06 agent-name dockerd[1224]: time="2023-07-04T17:59:06.472907976+03:00" level=error msg="restartmanger wait error: driver failed programming external connectivity on endpoint app-name (param): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))"' timestamp: 'Jul 4 17:59:06' hostname: 'agent-name' program_name: 'dockerd' **Phase 2: Completed decoding. name: 'docker' docker.level: 'error' docker.message: 'restartmanger wait error: driver failed programming external connectivity on endpoint app-name (param): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))' **Phase 3: Completed filtering (rules). id: '86003' level: '3' description: 'Docker: Error message' groups: '['docker', 'docker-error']' firedtimes: '1' mail: 'False' **Alert to be generated. Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.504134355+03:00" level=warning msg="Failed to allocate and map port 8000-8000: (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))" **Phase 1: Completed pre-decoding. full event: 'Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.504134355+03:00" level=warning msg="Failed to allocate and map port 8000-8000: (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))"' timestamp: 'Jul 4 17:59:07' hostname: 'agent-name' program_name: 'dockerd' **Phase 2: Completed decoding. name: 'docker' docker.level: 'warning' docker.message: 'Failed to allocate and map port 8000-8000: (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))' **Phase 3: Completed filtering (rules). id: '86002' level: '1' description: 'Docker: Warning message' groups: '['docker', 'docker-warning']' firedtimes: '1' mail: 'False' Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.532416050+03:00" level=error msg="cleanup: failed to delete container from containerd: container \"\" in namespace \"\": not found" **Phase 1: Completed pre-decoding. full event: 'Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.532416050+03:00" level=error msg="cleanup: failed to delete container from containerd: container \"\" in namespace \"\": not found"' timestamp: 'Jul 4 17:59:07' hostname: 'agent-name' program_name: 'dockerd' **Phase 2: Completed decoding. name: 'docker' docker.level: 'error' docker.message: 'cleanup: failed to delete container from containerd: container \' **Phase 3: Completed filtering (rules). id: '86003' level: '3' description: 'Docker: Error message' groups: '['docker', 'docker-error']' firedtimes: '2' mail: 'False' **Alert to be generated. Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.651138519+03:00" level=error msg="restartmanger wait error: driver failed programming external connectivity on endpoint app (): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))" **Phase 1: Completed pre-decoding. full event: 'Jul 4 17:59:07 agent-name dockerd[1224]: time="2023-07-04T17:59:07.651138519+03:00" level=error msg="restartmanger wait error: driver failed programming external connectivity on endpoint app (): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))"' timestamp: 'Jul 4 17:59:07' hostname: 'agent-name' program_name: 'dockerd' **Phase 2: Completed decoding. name: 'docker' docker.level: 'error' docker.message: 'restartmanger wait error: driver failed programming external connectivity on endpoint app (): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport <> -j DNAT --to-destination <> ! -i app: (fork/exec /sbin/iptables: no such file or directory))' **Phase 3: Completed filtering (rules). id: '86003' level: '3' description: 'Docker: Error message' groups: '['docker', 'docker-error']' firedtimes: '3' mail: 'False' **Alert to be generated.Hope this helps you,
Luis.
Sofia Amfiteatrova
Sofia Amfiteatrova
<program_name>^dockerd|^containerd</program_name>
<prematch>^time="\d+-\d+-\d+T\d+:\d+:\d+.\d+\+.\d:\d+." level=\S+ msg="</prematch>
<regex>level=(\S+) msg="(\.+)"</regex>
<order>docker.level,docker.message</order>
</decoder>
Luis González Romero
If you want to check the log that way, you have to remove the header (timestamp name->location). For example:
Having this log
The full log that wazuh will use is the one that follows 2023 Jul 04 14:59:07 (agent-name) any->/var/log/syslog:
Luis González Romero
It is strange, it works for me also with in the ruleset tool.
Are you sure that you added the decoder? You can check it by navigating to Management->Decoders and then you can search for docker
Could you please share the result for you? So I can check the patterns you have. Do not forget to save the local_decoder changes so the decoder test feature works with the new changes. And if you want that these changes take effect with the alerts you have to restart the manager. If you just want to test some patterns/regex you don't need to restart, but if you want to make them permanent you have to.
Luis González Romero
Sofia Amfiteatrova
Luis González Romero
The dashboard graphs are not related to the new decoders. Besides, the logall option takes a high amount of memory, you should disable it when you finish the debugging and all is working.
There are some error events in the third image. Do you mean that there are more events within your archives/alerts and they don’t show up within the events? In that case, can you share the output for these commands?
- cat /var/ossec/logs/alerts/alerts.json | grep docker
- cat /var/ossec/logs/alerts/alerts.json | grep docker
About the dashboard, do you mean that docker events does not appear within the graphs? That’s not related to the decoders. You should ask in another thread about the Top 5 events graph to give visibility, but maybe you can try to forward more events and check that graph.
Sofia Amfiteatrova
Luis González Romero
I suppose that the logs have rotated, because you shared a picture with two error alerts. And if the events appear within the dashboard, they do in the alerts files. How are you forwarding the docker logs? because they were working before. Did you follow this guide within the documentation? https://documentation.wazuh.com/current/user-manual/capabilities/container-security/monitoring-docker.html
Sofia Amfiteatrova
Luis González Romero
I think I’m missing something. Could you describe what do you want to achieve?
When you say
There are no options on this article for logs
You are able to see them within the Events section that you shared, where the error event appears. (You had two events within your events)
And I assume that you want something as the docker listener article, where you have some graphs for each monitored command (get, post, etc.). Did I miss something?
Sofia Amfiteatrova
