Custom decoder not recognized

[rwag...@gmail.com]

Hey guys!
I made a decoder for pfSense, but it is not being recognized by ossec.

Follow the decoder with a log sample:

<!-- Nov  7 12:37:34 pfSense filterlog: 5,,,1000102433,em0,match,block,in,4,0x0,,128,24677,0,none,17,udp,186,10.9.0.119,10.9.0.255,17500,17500,166 -->
<decoder name="pfsense">
  <program_name>pfsense</program_name>
</decoder>

<decoder name="pfsense">
    <prematch>^\w+  \d+ \d+:\d+:\d+ pfSense |\w+ \d+ \d+:\d+:\d+ pfSense </prematch>
</decoder>

<decoder name="pfsense_filter">
  <parent>pfsense</parent>
  <regex offset="after_parent">^filterlog: \d+,,,\d+,\S+,\w+,\w+,\w+,\d+,\S+,,\d+,\d+,\d+,none,\d+,udp,\d+,(\d+.\d+.\d+.\d+),(\d+.\d+.\d+.\d+),\d+,\d+,\d+$</regex>
  <order>srcip, dstip</order>
</decoder>

I put it in the folder of the decoders and tested with the ossec-logtest, follow the output:





 I'd like to know what's wrong.

[dan (ddp)]

> I'd like to know what's wrong.
>

Here is the output of the log sample before adding a decoder:
**Phase 1: Completed pre-decoding.
full event: 'Nov 7 12:37:34 pfSense filterlog:
5,,,1000102433,em0,match,block,in,4,0x0,,128,24677,0,none,17,udp,186,10.9.0.119,10.9.0.255,17500,17500,166'
hostname: 'pfSense'
program_name: 'filterlog'
log: '5,,,1000102433,em0,match,block,in,4,0x0,,128,24677,0,none,17,udp,186,10.9.0.119,10.9.0.255,17500,17500,166'

**Phase 2: Completed decoding.
No decoder matched.


Pay close attention to the "log:" line. Everything before that in the
log message won't be parsed by decoders. It's metadata, and it's taken
care of in pre-decoding. You can see the hostname and program there in
Phase 1.

Let's try your first decoder:
<decoder name="pfsense">
<program_name>^filterlog</program_name>
</decoder>

Adding this simple decoder to `/var/ossec/etc/local_decoder.xml` gives
us the following output:
**Phase 1: Completed pre-decoding.
full event: 'Nov 7 12:37:34 pfSense filterlog:
5,,,1000102433,em0,match,block,in,4,0x0,,128,24677,0,none,17,udp,186,10.9.0.119,10.9.0.255,17500,17500,166'
hostname: 'pfSense'
program_name: 'filterlog'
log: '5,,,1000102433,em0,match,block,in,4,0x0,,128,24677,0,none,17,udp,186,10.9.0.119,10.9.0.255,17500,17500,166'

**Phase 2: Completed decoding.
decoder: 'pfsense'

So now Phase 2 matches the pfsense decoder, but we want more. We can
take one of your other decoders, remove some bits the decoder doesn't
see, and get some useful information. Here's what I ended up with:

<decoder name="pfsense_filter">
<parent>pfsense</parent>

<regex offset="after_parent">^\d+,,,\d+,\S+,\w+,\w+,\w+,\d+,\S+,,\d+,\d+,\d+,none,\d+,udp,\d+,(\d+.\d+.\d+.\d+),(\d+.\d+.\d+.\d+),\d+,\d+,\d+$</regex>

<order>srcip, dstip</order>
</decoder>

In your decoder the <regex> started with '^filterlog', but as we see
from the 'log:' entry in the logtest output, the decoders do not see
that information. It's covered in the first decoder which is looking
for the program_name which is handled by the pre-decoder (easy,
right?).
Here's the output I get after adding this decoder to local_decoder.xml:
**Phase 1: Completed pre-decoding.
full event: 'Nov 7 12:37:34 pfSense filterlog:
5,,,1000102433,em0,match,block,in,4,0x0,,128,24677,0,none,17,udp,186,10.9.0.119,10.9.0.255,17500,17500,166'
hostname: 'pfSense'
program_name: 'filterlog'
log: '5,,,1000102433,em0,match,block,in,4,0x0,,128,24677,0,none,17,udp,186,10.9.0.119,10.9.0.255,17500,17500,166'

**Phase 2: Completed decoding.
decoder: 'pfsense'
srcip: '10.9.0.119'
dstip: '10.9.0.255'

We now have src and dst IPs.


> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ossec-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ossec-list+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[rwag...@gmail.com]

Hi Dan!

I'm ashamed after your explanation, thank you very much for the answer. But I will still expose another problem I had trying to make this decoder. I opened a call on github

I think you have a problem with the pre-decoder. He is ignoring p "\d+" in "^\w\w\w  \d+". That's why the "log:" entry is starting with "5,,,1000102433,at0...".

If a 2-digit log entry is placed on the day, the "log:" entry will start correctly. The "\d+" should serve with one or more numerical inputs.

Here is a print of the problem and the github call:


https://github.com/ossec/ossec-hids/issues/1315

[dan (ddp)]

On Nov 9, 2017 14:01, <rwag...@gmail.com> wrote:

Hi Dan!

I'm ashamed after your explanation, thank you very much for the answer. But I will still expose another problem I had trying to make this decoder. I opened a call on github

I think you have a problem with the pre-decoder. He is ignoring p "\d+" in "^\w\w\w  \d+". That's why the "log:" entry is starting with "5,,,1000102433,at0...".

If a 2-digit log entry is placed on the day, the "log:" entry will start correctly. The "\d+" should serve with one or more numerical inputs.

It's hard to tell feom a picture, but I think there are too many spaces between Nov and 18. There should only be 1 space in double digit dates, not 2. 2 spaces confuses the pre-decoder, so everything gets decoded instead of just the log.

...

[rwag...@gmail.com]

So, there are 2 spaces between the "MMM" and the day, but it's the pfSense log, it's like this. And the problem is in have 1 digit, when it has 2 digits the problem does not occur.

2 spaces confuses the pre-decoder

And so anyway is it really a bug?

I also tested as follows(whitout sucess):

<prematch>^\w\w\w\s\s\d+ \d\d:\d\d:\d\d pfSense</prematch>

[dan (ddp)]

On Thu, Nov 9, 2017 at 5:12 PM, <rwag...@gmail.com> wrote:
> So, there are 2 spaces between the "MMM" and the day, but it's the pfSense
> log, it's like this. And the problem is in have 1 digit, when it has 2
> digits the problem does not occur.
>
>
>> 2 spaces confuses the pre-decoder
>
> And so anyway is it really a bug?

Pretty much every log message I've seen follows the same rule (2
spaces for single digit days, 1 space for double digit days). I'd say
it's a bug in pfsense logs.

>
> I also tested as follows(whitout sucess):
>
> <prematch>^\w\w\w\s\s\d+ \d\d:\d\d:\d\d pfSense</prematch>
>

The problem is the inconsistency in the pfsense logs. You could try
modifying the predecoding parts of the code to handle their strange
logs, but I'd be worried about fallout from such changes.

>
> Em quinta-feira, 9 de novembro de 2017 16:25:11 UTC-3, dan (ddpbsd)
> escreveu:
>>
>>
>>
>> On Nov 9, 2017 14:01, <rwag...@gmail.com> wrote:
>>
>> Hi Dan!
>>
>> I'm ashamed after your explanation, thank you very much for the answer.
>> But I will still expose another problem I had trying to make this decoder. I
>> opened a call on github
>>
>> I think you have a problem with the pre-decoder. He is ignoring p "\d+" in
>> "^\w\w\w \d+". That's why the "log:" entry is starting with
>> "5,,,1000102433,at0...".
>>
>> If a 2-digit log entry is placed on the day, the "log:" entry will start
>> correctly. The "\d+" should serve with one or more numerical inputs.
>>
>>
>>
>> It's hard to tell feom a picture, but I think there are too many spaces
>> between Nov and 18. There should only be 1 space in double digit dates, not
>> 2. 2 spaces confuses the pre-decoder, so everything gets decoded instead of
>> just the log.
>>
>>
>>
>> Here is a print of the problem and the github call:
>>
>>

[rwag...@gmail.com]

Hello Dan!
I was wrong, when the log has 2 digits in the day field, there's only one space, the way you said it, sorry.
But I still have a problem, as the date is as metadata, how do I decode it as timestamp?

See in the entry below from the kibana, that the date field is not recognized as timestamp, I sent an entry from another date, to test and that was the output:





How do solve this?