need help with parsing nestat output with custom decoder and rule

[Oleksandr Kolesnyk]

Hi all, 

I want to parse netstat output for outgoing connections and have IP, port, protocol in separate fields. 

It works with logtest, but in kibana I don't see IP,port protocol - variables are empty. 

Would you please help me with it? 

My agent config is: 

  <localfile>
    <log_format>full_command</log_format>
    <command>netstat -nputw | sed 1,2d| tr -s ' ' | cut -f1,5,6 -d ' ' | grep -v '0.0.0.0:*\|127.0.0.1\|192.168.1\|172.18.0\| 172.19.0\|10.\|:::*'| grep 'ESTABLISHED' | sort
</command>
    <frequency>30</frequency>
    <alias>netstat outbound connections to external IPs</alias>
  </localfile>

My decoder is: 

<decoder name="netstat_ext_connections">
        <parent>ossec</parent>
        <program_name>netstat_ext_connections</program_name>
        <use_own_name>true</use_own_name>
        <prematch>ossec: output: 'netstat outbound connections to external IPs'</prematch>
        <regex offset="after_prematch">\\n(\w+) (\S+):(\d+) (\w+)</regex>
        <order>protocol,dstip,dst_port,connection_status</order>
</decoder>

My rules are:

  <rule id="102017" level="9">
        <decoded_as>netstat_ext_connections</decoded_as>
    <description>Connection to external IP $(dstip) and port $(dst_port) $(connection_status) $(protocol) on backend server.</description>
    <group>pci_dss_10.2.7,pci_dss_10.6.1,gpg13_10.1,gdpr_IV_35.7.d,</group>
  </rule>

My output in logtest is: 

ossec: output: 'netstat outbound connections to external IPs':\ntcp 52.208.61.52:443 ESTABLISHED

**Phase 1: Completed pre-decoding.

       full event: 'ossec: output: 'netstat outbound connections to external IPs':\ntcp 52.208.61.52:443 ESTABLISHED'

       timestamp: '(null)'

       hostname: 'manager01'

       program_name: '(null)'

       log: 'ossec: output: 'netstat outbound connections to external IPs':\ntcp 52.208.61.52:443 ESTABLISHED'

**Phase 2: Completed decoding.

       decoder: 'ossec'

       protocol: 'tcp'

       dstip: '52.208.61.52'

       dst_port: '443'

       connection_status: 'ESTABLISHED'

**Phase 3: Completed filtering (rules).

       Rule id: '102017'

       Level: '9'

       Description: 'Connection to external IP 52.208.61.52 and port 443 ESTABLISHED tcp on backend server.'

**Alert to be generated.

But in Kibana all variables are empty:

full_log	ossec: output: 'netstat outbound connections to external IPs': tcp 52.208.61.52:443 ESTABLISHED
	id	1587649399.69065270
	input.type	log
	location	netstat outbound connections to external IPs
	manager.name	manager01
	rule.description	Connection to external IP and port on backend server.

My log in archives.log:

2020 Apr 23 15:48:46 (agent01) 10.0.0.1->netstat outbound connections to external IPs ossec: output: 'netstat outbound connections to external IPs':

tcp 52.208.61.52:443 ESTABLISHED

My log in archives.json

{"timestamp":"2020-04-23T15:48:46.403+0200","rule":{"level":9,"description":"Connection to external IP  and port    on backend server.","id":"102017","firedtimes":24,"mail":false,"groups":["iw_custompci_dss_10.2.7"],"pci_dss":["10.6.1"],"gpg13":["10.1"],"gdpr":["IV_35.7.d"]},"agent":{"id":"093","name":"agent01","ip":"10.0.0.1"},"manager":{"name":"manager01"},"id":"1587649726.72144045","full_log":"ossec: output: 'netstat outbound connections to external IPs':\ntcp 52.208.61.52:443 ESTABLISHED","decoder":{"parent":"ossec","name":"netstat_ext_connections"},"location":"netstat outbound connections to external IPs"}

Thanks in advance!
Oleksandr

[Jesus Linares]

Hi,

I didn't have time to test it, but I recommend to change the regex to see where is the issue:

<prematch>ossec: output: 'netstat outbound connections to external IPs'</prematch>

<regex offset="after_prematch">(^.+)</regex>

In this way, you can see where is the "regex pointer".

Another tests that you can do: 

1. Check if the error is in the beginning:

 <regex offset="after_prematch">(tcp) (\S+):(\d+) (\w+)</regex>

 

2. Check if the error is in the end:

<regex offset="after_prematch">\\n(\w+) (\S+):(\d+) ESTABLISHED</regex>

3. Check the middle part:

<regex offset="after_prematch">(tcp) (\S+):(\d+) ESTABLISHED</regex>

 

Regards.

[Oleksandr Kolesnyk]

HI Jesus, 

It looks like that it brakes on the new line character. 

When I use the wildcard: 

<regex offset="after_prematch">^(\.+)</regex>

It parses the part with newline character in the logtest:

protocol: '\ntcp 34.248.76.180:443 ESTABLISHED'

and I see a parsed field in the alerts.json:

"data":{"protocol_":"\ntcp 34.248.76.180:443 ESTABLISHED"}

But if I try to move newline character out of the group:

<regex offset="after_prematch">^\\n(\.+)</regex>

or to the prematch: 

<prematch>ossec: output: 'netstat outbound connections to external IPs':\\n</prematch>

It still works in logtest, but it stops to work in the main application and I don't see parsed fields in the alerts.json

Regards,

Oleksandr

[Jesus Linares]

Hi,

Something weird is happening with the \\n character.

Could you try if some of the following decoders work:

1.

<decoder name="netstat_ext_connections">
        <parent>ossec</parent>
        <program_name>netstat_ext_connections</program_name>
        <use_own_name>true</use_own_name>

        <prematch>ossec: output: 'netstat outbound connections to external IPs':\.\.</prematch>
        <regex offset="after_prematch">(\S+) (\S+):(\d+) (\w+)</regex>
        <order>protocol,dstip,dst_port,connection_status</order>
</decoder>

2.

<decoder name="netstat_ext_connections">
        <parent>ossec</parent>
        <program_name>netstat_ext_connections</program_name>
        <use_own_name>true</use_own_name>
        <prematch>ossec: output: 'netstat outbound connections to external IPs':</prematch>

        <regex offset="after_prematch">(tcp) (\S+):(\d+) (\w+)|(\S+) (\S+):(\d+) (\w+)</regex>
        <order>protocol,dstip,dst_port,connection_status</order>
</decoder>

I hope it helps.

[Jesus Linares]

*Note that I changed both prematch and regex.

<button type="button" style="background-image:none;background-position:initial;background-repeat:initial;border-width:initial;border-style:none;border-color:initial;padding:4px;margin:0px;outline:none;color:rgb(0,107,180);border-radius:4px;vertical-align:middle;letter-spacing:-0.005em;max-width:100%;min-height:auto;

[Oleksandr Kolesnyk]

Hi Jesus,

Yes, it's strange behavior with a newline character. 

Finally, the regex that works for me is: 

<prematch>ossec: output: 'netstat outbound connections to external IPs':\.</prematch>
<regex offset="after_prematch">^(\S+) (\S+):(\d+) (\w+)</regex>

When I put \.\. in the prematch, then the first group in regex is 'cp' or 'dp', I concluded that the newline character is parsed as one character. 

Oleksandr

[Jesus Linares]

OK. It makes sense. I think the extra slash (\\n) was confused.

Checking the log and the prematch/regex is working as expected: \. capturing only 1 character (\n):

ossec: output: 'netstat outbound connections to external IPs':\ntcp 52.208.61.52:443 ESTABLISHED

<prematch>ossec: output: 'netstat outbound connections to external IPs':\.</prematch>
<regex offset="after_prematch">^(\S+) (\S+):(\d+) (\w+)</regex>

I'm glad that it is working.