Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Check Next Line For Certain Value or Existence

32 views
Skip to first unread message

wlamb...@gmail.com

unread,
Jun 12, 2018, 1:21:22 PM6/12/18
to
Hello,

I am fairly new to tcl and was hoping I could get some advice/tips on the best way to go about this.

I'm trying to parse through the data below using the below proc (already existing in tcl script). The whole script tails a log file and reads in each line, the below proc looks at each line to see if they match certain circumstances -- if they do not, they are added to the message payload and an event is sent to a daemon to be processed when it encounters a blank line (SendAlert proc).

This proc originally worked fine when a single blank line was used to signify the end of the log message, however, now, with the inclusion of another blank line, the whole message payload is not sent for all log entries. Of course, I could use SendAlert whenever I see the "**Alert" line, but that would cause one event to not be sent until another was written to the log file.


proc ProcessData { line } {
global AGENT_TYPE
global nDate sig_id priority message src_ip hexPayload payload agent src_port dst_port

# We do not care about the first line of an alert so find it and forget it
if { [regexp {(?x) ^\*\*\s+Alert} $line] } {
SendAlert
# nothing to do as we ignore this line

# See if this looks like a date line (the alert header, not the
# syslog line, which also begins with a date)
# look for logs sent by a OSSEC agent that should be treated as syslogs; e.g. Malwarebytes logs
} elseif { ([regexp {(?x)
# ^(\d\d\d\d)\s+(...)\s+(\d\d)\s+(\d\d:\d\d:\d\d)\s+(.*)->
# ^(\d\d\d\d)\s+(...)\s+(\d\d)\s+(\d\d:\d\d:\d\d)\s+(\(.*\)\s+)*(.*)->
^(\d\d\d\d)\s+(...)\s+(\d\d)\s+(\d\d:\d\d:\d\d)\s+(\(.*\)\s+)*(\S+)->(\d+.\d+.\d+.\d+)*
} $line MatchVar year month day time placeholder agent syslog_source]) } {
set nDate [clock format [clock scan "$day $month $year $time" ] -gmt true -f "%Y-%m-%d %T"]
# Ok, this is confusing, but the regexp can return either one
# or two variables, depending on the format of the input line.
# if the line is from a windows agent, it will usually contain
# "(host) X.X.X.X", but if it's from some other agent, it will
# usually just be one field (either a hostname or an IP address,
# depending on the log source). In either case, the $agent
# variable ends up holding the correct value for our purposes.
# SyslogSource will pull out the IP of the device sending a
# syslog to OSSEC.
if {[string length $syslog_source] != 0} {
set agent $syslog_source
} else {
set agent [ResolveHostname $agent]
}
} elseif { [regexp {(?x)
^Rule:\s+(\d+)\s+\(level\s+(\d+)\)\s+->\s+'(.*)'
} $line MatchVar sig_id priority message ] } {
set message "\[[string toupper $AGENT_TYPE]\] $message"
} elseif { [regexp {(?x) ^Src\s+IP:\s+(\S+)} $line MatchVar src_ip ] } {
set src_ip [ResolveHostname $src_ip]
} elseif { [regexp {(?x) ^Src\s+Port:\s+(\d+)} $line MatchVar src_port ] } {
# nothing to do as regexp filled the var
} elseif { [regexp {(?x) ^Dst\s+IP:\s+(\S+)} $line MatchVar agent ] } {
set agent [ResolveHostname $agent]
} elseif { [regexp {(?x) ^Dst\s+Port:\s+(\d+)} $line MatchVar dst_port ] } {
# nothing to do as regexp filled the var
# check to see if this is a blank line
# if it is then we've reached the end of the alert and can now send it to Sguil
} elseif { [regexp {(?x) ^\s*$} $line] } {
SendAlert
# If we haven't matched anything specific in the OSSEC alert
# structure, this must be a copy of the original alert.
# Add it to our payload.
} else {
append payload "$line\n"
}
}

----------

Data:

** Alert 1528724208.1445: - ossec,syscheck,pci_dss_11.5,gpg13_4.11,gdpr_II_5.1.f,
2018 Jun 11 13:36:48 wazuh-docker->syscheck
Rule: 550 (level 7) -> 'Integrity checksum changed.'
Integrity checksum changed for: '/etc/hostname'
<this is a blank line>
File: /etc/hostname
New size: 22
New permissions: 100644
New user: root (0)
New group: root (0)
New MD5: 95917fe7436d049af07b5e133389dfbb
New SHA1: c189919888285e9a6c24680b10dcc779d0470fc2
Old date: Mon Jun 11 13:31:50 2018
New date: Mon Jun 11 13:35:03 2018
New inode: 12197
<this is a blank line>

Would anyone be able to offer any thoughts or advice on how to check if the line after the first blank line exists or contains "**Alert", so that I would know to either continue reading the entry or go ahead and send the message?

Thanks,
Wes

wlamb...@gmail.com

unread,
Jun 12, 2018, 1:49:09 PM6/12/18
to
Apologies. This part should have SendAlert remarked:

if { [regexp {(?x) ^\*\*\s+Alert} $line] } {
#SendAlert

heinrichmartin

unread,
Jun 12, 2018, 2:37:13 PM6/12/18
to
On Tuesday, June 12, 2018 at 7:21:22 PM UTC+2, wlamb...@gmail.com wrote:
> This proc originally worked fine when a single blank line was used to signify the end of the log message, however, now, with the inclusion of another blank line, the whole message payload is not sent for all log entries. Of course, I could use SendAlert whenever I see the "**Alert" line, but that would cause one event to not be sent until another was written to the log file.

Reading your text only, not the code, it seems you are transforming input - starting with a line "** Alert " and including one, but not two blank lines - to some output. I guess that your problem is *not* related to Tcl. Think of your requirements and design an algorithm to perform your task.

I am happy to help with hints about Tcl, when you have dedicated questions.

jda...@gmail.com

unread,
Jun 12, 2018, 3:14:51 PM6/12/18
to

I'm assuming the only reason not to SendAlert after reading the next ** Alert line is due to a possible time delay waiting for the next message.

Consider adding a timer so SendAlert is called 50 ms (or other convenient time after the ** Alert line is read. Then canceling the timer and sending pending data if a new ** Alert line comes in sooner:


> if { [regexp {(?x) ^\*\*\s+Alert} $line] } {
# if timer is active (after info fails if timer id not valid)
if {![catch {after info $::alertTimer}]} {
after cancel $::alertTimer
> SendAlert
}
# schedule sending this alert after a reasonable time delay
set ::alertTimer [after 50 SendAlert]
> # nothing to do as we ignore this line

Then removing other calls to SendAlert

The [after info $::alertTimer] fails if the timer is not active, or if the variable is not defined (first call). In either case you do not need to SendAlert

Dave B

wlamb...@gmail.com

unread,
Jun 12, 2018, 3:29:46 PM6/12/18
to
Hi Dave,

Thanks for your response. I may not have conveyed it well, but if I use the next "**Alert" line to signify the end of the alert in the log, it could be minutes, hours, or days before another alert is populated, so I would be receiving critical alerts at a later date than they were actually populated into the log file.

I would need some method of determining if I have read the whole log entry (either with one or two spaces in it) and be able to add the necessary data to the message payload and send that off to another process.

I was thinking if I could store the previous line and compare it with the current (say, if the previous line was a blank line, and the current is not a line containing **Alert or is not a blank line or doesn't exist , then continue adding to the payload until we are able to go to the next blank line and apply the same logic and finally send an alert).

I agree with the first response, in that, maybe I need to really think about what I am trying to do -- I'm just trying to get some advice on the best approach.

Thanks,
Wes

jda...@gmail.com

unread,
Jun 12, 2018, 3:51:14 PM6/12/18
to

> Thanks for your response. I may not have conveyed it well, but if I use the next "**Alert" line to signify the end of the alert in the log, it could be minutes, hours, or days before another alert is populated, so I would be receiving critical alerts at a later date than they were actually populated into the log file.
>

I'm assuming the ** Alert line marks the start of an alert message in the log file. I'm suggesting you SendAlert 50 milliseconds after the **Alert line appears in the log file (giving enough time for the rest of the message to be oarsed from the log file), or when a new ** Alert line appears (starting a new message). Using this scheme an alert would be sent at most 50 ms after it appeared in the log file, even if no new ones appear in the log file for hours or days.

Dave B

wlamb...@gmail.com

unread,
Jun 12, 2018, 4:11:10 PM6/12/18
to
Thanks, Dave. To be clear, I would not check for a blank line at all then, but would just add every other line to the payload, until I hit a line with **Alert, or until 50ms have passed -- is that correct?

Thanks,
Wes

jda...@gmail.com

unread,
Jun 12, 2018, 4:17:40 PM6/12/18
to
yes, that is what I'm suggesting.

The delay time needs to be adjusted to match your system.

I'm not sure what happens in SendAlert, but you might want to clear all the global variables so no old data gets included in the next message.

Dave B

Rich

unread,
Jun 12, 2018, 4:22:51 PM6/12/18
to
wlamb...@gmail.com wrote:
>
> I am fairly new to tcl and was hoping I could get some advice/tips on
> the best way to go about this.

We may be able to offer some help, but as Heinrich said, you don not
have a Tcl problem, you have an algorithm (unrelated to any programming
language) problem.

> I'm trying to parse through the data below using the below proc
> (already existing in tcl script). The whole script tails a log file
> and reads in each line, the below proc looks at each line to see if
> they match certain circumstances -- if they do not, they are added to
> the message payload and an event is sent to a daemon to be processed
> when it encounters a blank line (SendAlert proc).
>
> This proc originally worked fine when a single blank line was used to
> signify the end of the log message, however, now, with the inclusion
> of another blank line, the whole message payload is not sent for all
> log entries.

Ok, so, first, simplest, solution... Is there any way you can have the
"second blank line" removed from the log so that what the proc expects
is what the proc receives?

> Of course, I could use SendAlert whenever I see the "**Alert" line,
> but that would cause one event to not be sent until another was
> written to the log file.
> ----------
>
> Data:
>
> ** Alert 1528724208.1445: - ossec,syscheck,pci_dss_11.5,gpg13_4.11,gdpr_II_5.1.f,
> 2018 Jun 11 13:36:48 wazuh-docker->syscheck
> Rule: 550 (level 7) -> 'Integrity checksum changed.'
> Integrity checksum changed for: '/etc/hostname'
> <this is a blank line>
> File: /etc/hostname
> New size: 22
> New permissions: 100644
> New user: root (0)
> New group: root (0)
> New MD5: 95917fe7436d049af07b5e133389dfbb
> New SHA1: c189919888285e9a6c24680b10dcc779d0470fc2
> Old date: Mon Jun 11 13:31:50 2018
> New date: Mon Jun 11 13:35:03 2018
> New inode: 12197
> <this is a blank line>
>
> Would anyone be able to offer any thoughts or advice on how to check
> if the line after the first blank line exists or contains "**Alert",
> so that I would know to either continue reading the entry or go ahead
> and send the message?

You are thinking about this wrong. You are "tailing" a log file,
therefore as you said in another reply, you may not get a "next" line
for hours or days, because nothing new arrived in the log file.

So you can't "look ahead" (which is what you are asking to do) in the
log file, because that which is "ahead" may not arrive for two weeks.

You can only operate upon the past with a log file. You know what you
have previously received, and what you now have, but you can't
reasonably do something with "future data" that has not yet arrived.

As well, that proc looks like a huge mess that does a lot of unused
work, to then concatenate lines of the log file into a message to this
deamon. Did you remove a lot of other stuff from it for posting it
here? If yes, then that is why. But if not, you might get a good
start by trimming out the unused work.

Next, is this "extra blank line" always present in new log records? So
a log record will begin with "** Alert" and will end after the second
blank line? Then why not look for a line starting with "** Alert",
then accumulate lines until you receive two blank lines, and as soon as
you receive the second blank line, send the accumulated lines as the
alert?

Alternately, if you have any control over the system generating these
log records, can that system be modified to output an explicit "end of
alert" line. I.e.:

** Alert ...
... multiple lines here
** End of Alert

With an explicitly defined end signal, chunking this into individual
records would be a trivial task.

wlamb...@gmail.com

unread,
Jun 12, 2018, 4:42:15 PM6/12/18
to
Thanks Rich,

One of the first things I wondered was if I could change the formatting of the log -- I am currently waiting on an answer for that.

Agreed, it's silly of me to try to compare to future data when it will not be available. I think I am just confusing myself by trying to overthink a simple solution.

The proc itself is from a larger tcl script, but that is the entirety of the ProcessData proc -- it sits in a while loop, looping over lines until the end of the file is reached.

Currently, an alert can consist of one or two blank lines, so it makes it difficult and somewhat impractical to try to use that to determine when the log ends.

You are correct in that a specific terminator would be ideal, however I am not aware of how to modify/implment this currently.

I'm thinking Dave;s timer approach might be the best option until I can figure out how to modify the alert output itself.

Thanks for your feedback.

Wes

heinrichmartin

unread,
Jun 12, 2018, 4:52:15 PM6/12/18
to
On Tuesday, June 12, 2018 at 10:42:15 PM UTC+2, wlamb...@gmail.com wrote:
> I'm thinking Dave;s timer approach might be the best option until I can figure out how to modify the alert output itself.

You mentioned that you are facing legacy code. For a greenfield approach you should consider Expect. It provides readable syntax for handling all the patterns and can implicitly perform the loop...

hth
Martin

Rich

unread,
Jun 12, 2018, 6:18:40 PM6/12/18
to
wlamb...@gmail.com wrote:
> One of the first things I wondered was if I could change the
> formatting of the log -- I am currently waiting on an answer for
> that.

This is, overall, the best solution (but as it obviously involves
others, it may not be an /available/ solution).

> Agreed, it's silly of me to try to compare to future data when it
> will not be available. I think I am just confusing myself by trying
> to overthink a simple solution.
>
> The proc itself is from a larger tcl script, but that is the entirety
> of the ProcessData proc -- it sits in a while loop, looping over
> lines until the end of the file is reached.

What is all the parsing of lines of the log, that all look to be
ignored, supposed to do?

> Currently, an alert can consist of one or two blank lines, so it
> makes it difficult and somewhat impractical to try to use that to
> determine when the log ends.

Yes, that makes detecting 'end of current record' difficult.

Does each log record always end with the same set of lines? If so you
might trigger your 'send' by noticing when you receive the last line of
an entry (assuming that last line is generally the same).

> You are correct in that a specific terminator would be ideal, however
> I am not aware of how to modify/implment this currently.

Another change by the 'others' to the log format. All they have to do
is output something that is a unique terminator line when they are done
outputting their records, and you've got something to look for. Plus,
it gives them freedom to tweak/adjust/change the interveaning lines all
they like without breaking the "collect and send alert" code.

> I'm thinking Dave;s timer approach might be the best option until I
> can figure out how to modify the alert output itself.

That may be all you have available if you can't get changes made.
0 new messages