Pattern works against test log, logs are being received on interface, ELSA not getting logs.

[fear nothing]

- I have a working pattern for the syslog messages that are being sent.
- If I use the pdbtool against my patterndb.xml and manually paste one of the messages there, it matches correctly.
- My syslog messages are being received by SO on the right port according to tcpdump.
- I have restarted the syslog-ng service.

But ELSA is not seeing any logs. Um, wat? What step have I missed out here?

[Doug Burks]

Hi fear.nothing,

When you go to https://security.onion/elsa and click Host Logs -
Syslog-NG (Host), is the sending host listed?

If so, drill into that sending host and see if you the logs you're
looking for are there.

If not, can you include your pattern and a log sample in your reply?

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[fear nothing]

No, the host is not listed.

extract from patterndb here: http://pastebin.com/3HFX1eWz

TCPdump output:

06:28:31.302775 IP (tos 0x0, ttl 64, id 35415, offset 0, flags [none], proto UDP (17), length 223)
host.domain.syslog > 192.168.1.5.syslog: SYSLOG, length: 195
Facility local0 (16), Severity info (6)
Msg: Mar 9 07:28:31 filterlog: 170,16777216,,1424282996,em1,match,block,in,4,0x0,,128,1063,0,DF,6,tcp,52,192.168.1.10,10.15.0.114,59652,7810,0,S,224713639,,8192,,mss;nop;wscale;nop;nop;sackOK
06:28:32.301250 IP (tos 0x0, ttl 64, id 36847, offset 0, flags [none], proto UDP (17), length 174)
host.domain.syslog > 192.168.1.5.syslog: SYSLOG, length: 146
Facility local0 (16), Severity info (6)
Msg: Mar 9 07:28:32 filterlog: 171,16777216,,1424282996,em1,match,block,in,4,0x0,,1,2042,0,none,17,udp,50,192.168.1.10,224.0.0.252,49813,5355,30

[Doug Burks]

Just to confirm...

Does the following command run with no errors?
pdbtool test /etc/elsa/patterns.d/local/my_new_log_parser

Did you merge your new pattern into /opt/elsa/node/conf/patterndb.xml
and can you see it there?

Have you confirmed that port 514 (udp and tcp) are open in UFW?
sudo ufw status

[fear nothing]

I've merged the pattern and when I run the test against patterndb.xml, the lines for my patterns produce no errors.

Port 514 is open, allow from anywhere.

[fear nothing]

OK, I'm getting somewhere. I started from scratch, complete fresh install of SO.

Immediately after configuring SO, ELSA started getting uncategorised logs from pfsense - I could load and see them as well as stuff from cron and sudo that I was never getting before. Great start!

So, I added my parser in. No dice.
If I run the command
'pdbtool pdbtool match -M "Mar 20 01:00:00 filterlog: 171,16777216,,1424282996,em1,match,block,in,4,0x0,,128,26102,0,none,17,udp,78,192.168.1.105,192.168.1.255,137,137,58" -P filterlog -p /opt/elsa/node/conf/patterndb.xml'
it prints the correct output. However if I don't specify the program, puts it under TAGS=.classifier.unknown

Does this tell you which part of the pattern I've gotten wrong?

[fear nothing]

Oh, and when I look at the content in ELSA web interface, it identifies the program correctly and says "class=NONE"

[BBCan177]

Hi fear nothing,

I haven't played with pfSense 2.2.x parsers yet for ELSA, but the previous ones would use class=FIREWALL_ACCESS_DENY for the Block Alerts.

Previous parsers here for 2.1.x:
http://www.securitygrit.com/2013/03/pfsense-into-elsa.html

[fear nothing]

Yes I found that before and based my rules off that example; the issue is that even though ELSA is tagging incoming logs with 'program=filterlog', it's not applying the parser I've written to them (see pastebin linked before http://pastebin.com/3HFX1eWz ). I'm sure it's just a tiny detail I've misinterpreted, but I can't see what it might be.

[fear nothing]

Good news folks: I've found what was wrong. In basic terms, I was being an idiot newbie. I'll be working on this a bit more to cover more than just the basic IPv4 cases but a full patterndb for you lovely folks is incoming shortly.

[fear nothing]

I haven't covered everything yet but you are welcome to the draft
version which is working for everything I've seen so far, barring ICMP6.
I've put the pattern up on pastebin here:
http://pastebin.com/9HFqVYWE
I will try to get ICMP6 done and tested soon, which will
leave CARP as the only case I know I haven't covered (there may be
others, and there might be a better way to write the cases which over
IGMP and Hop Options).

If anyone finds an edge case which these patterns don't catch please post the message content and I will try to incorporate it.

[Doug Burks]

Thanks!

pdbtool test reports:

Testing message program='filterlog'
message='141,16777216,,1424282984,em1,match,pass,in,4,0x0,,128,26522,0,DF,6,tcp,52,192.168.1.13,74.125.136.159,63408,443,0,S,942122724,,8192,,mss;nop;wscale;nop;nop;sackOK'
Testing message program='filterlog'
message='5,16777216,,1000000103,em0,match,pass,in,4,0x0,,41,63030,0,none,1,icmp,107,222.233.210.210,109.88.162.79,unreachport,222.233.210.210,UDP,5387'
Wrong match name='i0', value='1', expected='4'
Testing message program='filterlog'
message='140,16777216,,1424282984,em1,match,pass,in,4,0x0,,128,4207,0,DF,6,tcp,52,192.168.0.3,216.58.208.66,65480,80,0,S,452276625,,8192,,mss;nop;wscale;nop;nop;sackOK'
Testing message program='filterlog'
message='170,16777216,,1424282996,em0,match,block,in,4,0x0,,128,18835,0,DF,6,tcp,48,193.110.44.49,10.15.180.114,55253,25,0,S,2587172434,,8192,,mss;nop;nop;sackOK'
Testing message program='filterlog'
message='171,16777216,,1424282996,em1,match,block,in,4,0x0,,1,30898,0,none,17,udp,50,192.168.1.10,224.0.0.252,63227,5355,30'

so I'm changing the following example:

<example>

<test_messageprogram="filterlog">5,16777216,,1000000103,em0,match,pass,in,4,0x0,,41,63030,0,none,1,icmp,107,222.233.210.210,109.88.162.79,unreachport,222.233.210.210,UDP,5387</test_message>
<test_value
name="s1">em0</test_value>
<test_value
name="i0">4</test_value>
<test_value
name="i1">222.233.210.210</test_value>
<test_value
name="i2"></test_value>
<test_value
name="i3">109.88.162.79</test_value>
<test_value
name="i4"></test_value>
</example>

to:

<example>

<test_messageprogram="filterlog">5,16777216,,1000000103,em0,match,pass,in,4,0x0,,41,63030,0,none,1,icmp,107,222.233.210.210,109.88.162.79,unreachport,222.233.210.210,UDP,5387</test_message>
<test_value
name="s1">em0</test_value>
<test_value
name="i0">1</test_value>
<test_value
name="i1">222.233.210.210</test_value>
<test_value
name="i2"></test_value>
<test_value
name="i3">109.88.162.79</test_value>
<test_value
name="i4"></test_value>
</example>

Other than that, looks good! Great job!

[fear nothing]

Whoops, that's what happens when you copy/paste the test section instead of writing it out from fresh!