The space issue appears again
Digital X
Everything was working great until I upgraded to the git of liblognorm.
Here's the raw log via sagan -d syslog:
[*]
127.0.0.1|kern|warning|warning|kernel:|2011-11-05|22:33:28|kernel|[842455.0
50497] IN=ppp0 OUT= MAC= SRC=1.1.1.1 DST=1.1.1.1 LEN=48 TOS=0x00 PREC=0x00
TTL=115 ID=14317 PROTO=TCP SPT=3978 DPT=4899 WINDOW=65535 RES=0x00 SYN
URGP=0
From -d normalize:
[*] Normalize output: [cee@115 originalmsg="[842455.050497\] IN=ppp0 OUT=
MAC= SRC=1.1.1.1 DST=1.1.1.1 LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=14317
PROTO=TCP SPT=3978 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 "
unparsed-data="[842455.050497\] IN=ppp0 OUT= MAC= SRC=1.1.1.1 DST=1.1.1.1
LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=14317 PROTO=TCP SPT=3978 DPT=4899
WINDOW=65535 RES=0x00 SYN URGP=0 "]
I know my issue is the space before or after. Here's the rule, with a
space at the end:
rule=:%u-:word% IN=%-:word% %-:word% %-:word% SRC=%src-ip:ipv4%
DST=%dst-ip:ipv4% LEN=%-:number% TOS=%-:word% PREC=%-:word% TTL=%-:number%
ID=%-:number% PROTO=TCP SPT=%src-port:number% DPT=%dst-port:number%
WINDOW=%-:number% RES=%-:word% %-:word% URGP=%-:word%
Now, if I echo this:
'127.0.0.1|kern|warning|warning|kernel:|2011-11-05|22:33:28|kernel|
[842455.050497] IN=ppp0 OUT= MAC= SRC=1.1.1.1 DST=1.1.1.1 LEN=48 TOS=0x00
PREC=0x00 TTL=115 ID=14317 PROTO=TCP SPT=3978 DPT=4899 WINDOW=65535
RES=0x00 SYN URGP=0 '
with the the space between kernel| [842...
I get:
[*] Normalize output: [cee@115 originalmsg=" [842455.050497\] IN=ppp0 OUT=
MAC= SRC=1.1.1.1 DST=1.1.1.1 LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=14317
PROTO=TCP SPT=3978 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 "
unparsed-data=" "]
Lastly, removing the space at the end I get a full match:
[*] Normalize output: [cee@115 -="0" -="SYN" -="0x00" -="65535"
dst-port="4899" src-port="3978" -="14317" -="115" -="0x00" -="0x00" -="48"
dst-ip="1.1.1.1" src-ip="1.1.1.1" -="MAC=" -="OUT=" -="ppp0"
u-="[842455.050497\]"]
The lack of a space after kernel| is killing me. My workaround no longer
works...so what are my options? Thank you.
James
On 11/9/11 12:38 AM, "Rainer Gerhards" <rger...@hq.adiscon.com> wrote:
>> Very cool news...thanks Rainer. I've got the latest git installed and
>> up
>> and running. Regarding the -, would a sample look like say:
>>
>> %-:word%
>
>Yes, exactly :)
>
>rainer
>_______________________________________________
>Lognorm mailing list
>Log...@lists.adiscon.com
>http://lists.adiscon.net/mailman/listinfo/lognorm
cclark
> So....I've tried this out, and I think my issue is something
> different.
> Everything was working great until I upgraded to the git of
> liblognorm.
> Here's the raw log via sagan -d syslog:
>
> [*]
>
> 127.0.0.1|kern|warning|warning|kernel:|2011-11-05|22:33:28|kernel|[842455.0
> 50497] IN=ppp0 OUT= MAC= SRC=1.1.1.1 DST=1.1.1.1 LEN=48 TOS=0x00
> PREC=0x00
> TTL=115 ID=14317 PROTO=TCP SPT=3978 DPT=4899 WINDOW=65535 RES=0x00
> SYN
> URGP=0
This message doesn't look right. If you look at the Sagan FIFO (in
the syslog
software), it should have a space.. Like "| $MSG" (if using syslog-ng
for example).
It doesn't look like you have one.
>
> From -d normalize:
> [*] Normalize output: [cee@115 originalmsg="[842455.050497\] IN=ppp0
> OUT=
> MAC= SRC=1.1.1.1 DST=1.1.1.1 LEN=48 TOS=0x00 PREC=0x00 TTL=115
> ID=14317
> PROTO=TCP SPT=3978 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 "
> unparsed-data="[842455.050497\] IN=ppp0 OUT= MAC= SRC=1.1.1.1
> DST=1.1.1.1
> LEN=48 TOS=0x00 PREC=0x00 TTL=115 ID=14317 PROTO=TCP SPT=3978
> DPT=4899
> WINDOW=65535 RES=0x00 SYN URGP=0 "]
>
> I know my issue is the space before or after. Here's the rule, with
> a
> space at the end:
>
> rule=:%u-:word% IN=%-:word% %-:word% %-:word% SRC=%src-ip:ipv4%
> DST=%dst-ip:ipv4% LEN=%-:number% TOS=%-:word% PREC=%-:word%
> TTL=%-:number%
> ID=%-:number% PROTO=TCP SPT=%src-port:number% DPT=%dst-port:number%
> WINDOW=%-:number% RES=%-:word% %-:word% URGP=%-:word%
>
The rules _should_ have a space as well. IE:
rule=: %u-:word%
_not_
rule=:%u-:word%
> The lack of a space after kernel| is killing me. My workaround no
> longer
> works...so what are my options? Thank you.
>
Not sure as I haven't had time to test. I've been swamped with work,
but will
look into it ASAP. As a rule, however there needs to be a Space with
the FIFO
(in your syslog configuration "| $MSG") and the rule=: . I've talked
with Rainer about
this, and this is so it'll meet the syslog IETF/BSD specs (if i
remember correctly).
Hope this helps.
cclark
A while back, someone pointed out a interesting normalization issue.
Actually, it's not really a "normalization" issue, but more about how
the information is being displayed.
If Sagan is doing direct database writes, you'll notice in your
console of choice (Snorby/Prelude/BASE/whatever) events like this:
[OPENSSH] Authentication failure [root] [uid: 0]
Which is pretty nice.... However, with you're using Unified2 with
Barnyard2, events will look like this:
[OPENSSH] Authentication failure
Normalization is actually _still_ happening, but due to the way
unified2/barnyard2 work, Sagan can't append the "[username] [uid]" to
the message.
If Sagan is directly writing to the database, it is doing all the
functions that Barnyard2 is doing. Most importantly, updating the
"signatures" table. When you use unified2, Sagan loses that control.
Sagan simple "writes" the unified2 file with the signature ID (for
example 5000312) that was triggered. It is _barnyard2_ does
the lookup of the signature and inserts that events into the database
on Sagan behalf.
While the data is being normalized, Sagan + unified2 can't alter the
signature because Barnyard2 is doing on Sagan's behalf based on the
Signature ID triggered.
I'm trying to figure out a work around for this (with unified2).
I'm not seeing any way possible to accomplish this, and
this data is fairly important to the correlation process. Short of
giving Sagan the ability to "queue" events on it's own, I'm not seeing
a way around this with Barnyard2.
Let me know your thoughts.
--
Champ Clark III | Quadrant Information Security | 904-253-7856
http://www.quadrantsec.com
GPG Key ID: 0B30A6A7
Key fingerprint = A154 17D5 F16D 8C09 69FA 618B 3877 B04C 0B30 A6A7
If it wasn't for C, we'd be using BASI, PASAL and OBOL.
Digital X
Yea that's exactly the issue....the kernel messages do NOT have that space. My work workaround at the time was to set rule=:%...that worked, but now does not. The raw syslog doesn't have a space in it...so that's my issue. Not sure why..it's ONLY with kernel messages....every other one is fine.
James
Champ Clark III [Quadrant Information Security]
> Hey Champ,
>
> Yea that's exactly the issue....the kernel messages do NOT have that
> space. My work workaround at the time was to set rule=:%...that worked,
> but now does not. The raw syslog doesn't have a space in it...so that's my
> issue. Not sure why..it's ONLY with kernel messages....every other one is
> fine.
That's really, really confusing because the FIFO should be (for
syslog-ng for example):
template("$SOURCEIP|$FACILITY|$PRIORITY|$LEVEL|$TAG|$YEAR-$MONTH-$DAY|$HOUR:$MIN:$SEC|$PROGRAM| $MSG\n")
Note the "$PROGRAM| $MSG\n" <- That's where the space comes
from. Note the kernel message.
Champ Clark III [Quadrant Information Security]
>
> Yea that's exactly the issue....the kernel messages do NOT have that
> space. My work workaround at the time was to set rule=:%...that worked,
> but now does not. The raw syslog doesn't have a space in it...so that's my
> issue. Not sure why..it's ONLY with kernel messages....every other one is
> fine.
Ahh.. wow.. that's wierd and strange. What version/type of
syslog daemon are you using? Basically, if it's doing that, this has
to be a syslog daemon issue, because it's not following it's own
template!
kernel| [2749029.439633] grsec:
Champ Clark III [Quadrant Information Security]
> On Thu, Nov 10, 2011 at 02:01:46PM -0700, Digital X wrote:
> > Hey Champ,
> >
> > Yea that's exactly the issue....the kernel messages do NOT have that
> > space. My work workaround at the time was to set rule=:%...that worked,
> > but now does not. The raw syslog doesn't have a space in it...so that's my
> > issue. Not sure why..it's ONLY with kernel messages....every other one is
> > fine.
>
> Ahh.. wow.. that's wierd and strange. What version/type of
> syslog daemon are you using? Basically, if it's doing that, this has
> to be a syslog daemon issue, because it's not following it's own
> template!
>
> kernel| [2749029.439633] grsec:
I'm discussing this with DigitalX on IRC (irc.freenode.net
#sagan). We're pretty sure this is due to a old version of rsyslog.
This post was just added to keep the list up to date.
DigiAngel
On Nov 10, 2:34 pm, "Champ Clark III [Quadrant Information Security]"
> application_pgp-signature_part
> < 1KViewDownload