Parsing OSSEC XML Rules

[Dave Stoddard]

I am posting this to both the OSSEC and WAZUH mailing lists as this message applies to both. First, I want to express my appreciation for OSSEC and all of the work that the developers have put into this tool. I worked with a number of programs in the past to collect SIEM event data (Snare and others) and we had to write a lot of code to make it work. OSSEC is a superior application for event collection and rating for SIEM.

 

I have been working on a program to convert the OSSEC/WAZUH rules to a MySQL database table for use in an external monitoring application, and I thought I would share some of the things I discovered while going through this process. Note that none of these things are criticisms – it’s more of a road map for anyone that needs to convert the rules to another format. I used Perl with the LibXML library to do the parsing of the XML. Here are a few of the things I found:

 

1. The XML in the /var/ossec/rules directory is not conformant to strict XML standards in that the XML standard requires all elements in a file to be grouped within one outside element. In the OSSEC rules, many of the files contain multiple <group> elements on the first level, which breaks the LibXML parser. To get around this limitation, I slurped each file into a scalar variable and wrapped its contents in <outer> and </outer> elements. This solved the problem.

 

2. In the documentation for OSSEC rules, there are a number of elements that have been defined to function within a <rule> element container. I discovered that several elements are not defined in the documentation found here:

 

 http://ossec-docs.readthedocs.io/en/latest/syntax/head_rules.html

 

A list of all the elements used within a <rule> container for all of the files is provided below.  The elements that have an asterisk next to them are not defined in the current documentation:

 

action *

category

check_diff

check_if_ignored *

compiled_rule *

decoded_as

description

different_url *

extra_data

group

hostname

id

if_fts *

if_group

if_matched_group

if_matched_sid

if_sid

ignore *

info

match

options

program_name

regex

same_id

same_source_ip

same_user

status *

time

url

user

 

You can find some documentation on a few of these elements by doing a Google search, such as if_fts (first time seen), and compiled_rule. You can also grep the XML rule files to find other examples in use, however some of these elements are mysterious in their use including <action>, <check_if_ignored>, <ignore>, and <status>.

 

3. A number of elements have attributes, and I needed a definitive list in order to build a table that would capture those items. There are only four elements that use attributes in OSSEC rules. A list of those elements and their attributes is provided below:

 

<group> : name

<info> : type

<rule> : frequency, id, ignore, level, maxsize, noalert, overwrite, timeframe

<var> : name

 

Of these attributes, only the noalert attribute is undefined in the documentation.

 

4. One of the files, netscaler_rules.xml, has a comment with two hyphens in it (a sample log entry), which breaks XML (yes, the friggin LibXML parser scans comments!). I had to modify the comment by replacing one of the hyphens with an equal sign to allow it to continue.

 

5. When parsing the XML data, the structure of each file containing rules is very simple to process. There are no complex CDATA nodes to deal with, entity references, documents, or other esoteric data. The complete list of node types includes elements, attributes, text, comments, white space, and “end elements”. This level of simplicity makes it easy to read and process the rules into other formats.

 

If anyone is interested, I will post the code on the list after I get it working. We have a proprietary API we use to talk to our database, but I can rewrite those calls to make it more vanilla so others can use this too. Best,

 

Dave Stoddard

Network Alarm Corporation

 

[Pedro Sanchez]

Hi Dave, thanks for sharing your conclusions with us, let me answer you each point.

"convert the OSSEC/WAZUH rules to a MySQL database table"

In case it helps, I think OSSEC already insert each rule into a MySQL database, I can't try it out right now but you can see in the links below the MySQL schema and where OSSEC read and insert each rule.

https://github.com/wazuh/ossec-wazuh/blob/master/src/os_dbd/mysql.schema#L17

https://github.com/wazuh/ossec-wazuh/blob/5441889d963ce6d8ee3fae0e9f273e701b6c89eb/src/os_dbd/main.c#L218

1.

Totally agree, in fact, we had encounter this problem while creating the RESTful API, we use some Python parsers from XML to JSON.

We solved the problem adding a parent tag like you did, we are doing that in all XML levels, in some cases (for example, parsing ossec.conf) we introduced some particular solutions to each tag (studying all configuration possibilities).

2.

You are right, there are so many "hidden" options in OSSEC rules, we are working on document everything in detail.

3.

Same than 2.

4.

Agree, please feel free to send a pull request to our repository so it will be solved to everyone : D Thanks!

It sounds very interesting, please keep us updated on with your progress, we will be happy to help you.

Regarding to that API you mention, how is it going to work? will it ask the database for rules info?

Cheers,

Pedro S.

 

--

---
You received this message because you are subscribed to the Google Groups "ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.