Parsing OSSEC XML Rules

[Dave Stoddard]

I am posting this to both the OSSEC and WAZUH mailing lists as this message applies to both. First, I want to express my appreciation for OSSEC and all of the work that the developers have put into this tool. I worked with a number of programs in the past to collect SIEM event data (Snare and others) and we had to write a lot of code to make it work. OSSEC is a superior application for event collection and rating for SIEM.

 

I have been working on a program to convert the OSSEC/WAZUH rules to a MySQL database table for use in an external monitoring application, and I thought I would share some of the things I discovered while going through this process. Note that none of these things are criticisms – it’s more of a road map for anyone that needs to convert the rules to another format. I used Perl with the LibXML library to do the parsing of the XML. Here are a few of the things I found:

 

1. The XML in the /var/ossec/rules directory is not conformant to strict XML standards in that the XML standard requires all elements in a file to be grouped within one outside element. In the OSSEC rules, many of the files contain multiple <group> elements on the first level, which breaks the LibXML parser. To get around this limitation, I slurped each file into a scalar variable and wrapped its contents in <outer> and </outer> elements. This solved the problem.

 

2. In the documentation for OSSEC rules, there are a number of elements that have been defined to function within a <rule> element container. I discovered that several elements are not defined in the documentation found here:

 

 http://ossec-docs.readthedocs.io/en/latest/syntax/head_rules.html

 

A list of all the elements used within a <rule> container for all of the files is provided below.  The elements that have an asterisk next to them are not defined in the current documentation:

 

action *

category

check_diff

check_if_ignored *

compiled_rule *

decoded_as

description

different_url *

extra_data

group

hostname

id

if_fts *

if_group

if_matched_group

if_matched_sid

if_sid

ignore *

info

match

options

program_name

regex

same_id

same_source_ip

same_user

status *

time

url

user

 

You can find some documentation on a few of these elements by doing a Google search, such as if_fts (first time seen), and compiled_rule. You can also grep the XML rule files to find other examples in use, however some of these elements are mysterious in their use including <action>, <check_if_ignored>, <ignore>, and <status>.

 

3. A number of elements have attributes, and I needed a definitive list in order to build a table that would capture those items. There are only four elements that use attributes in OSSEC rules. A list of those elements and their attributes is provided below:

 

<group> : name

<info> : type

<rule> : frequency, id, ignore, level, maxsize, noalert, overwrite, timeframe

<var> : name

 

Of these attributes, only the noalert attribute is undefined in the documentation.

 

4. One of the files, netscaler_rules.xml, has a comment with two hyphens in it (a sample log entry), which breaks XML (yes, the friggin LibXML parser scans comments!). I had to modify the comment by replacing one of the hyphens with an equal sign to allow it to continue.

 

5. When parsing the XML data, the structure of each file containing rules is very simple to process. There are no complex CDATA nodes to deal with, entity references, documents, or other esoteric data. The complete list of node types includes elements, attributes, text, comments, white space, and “end elements”. This level of simplicity makes it easy to read and process the rules into other formats.

 

If anyone is interested, I will post the code on the list after I get it working. We have a proprietary API we use to talk to our database, but I can rewrite those calls to make it more vanilla so others can use this too. Best,

 

Dave Stoddard

Network Alarm Corporation

 

[Santiago Bassett]

Dave, thanks for the information. I already knew about some of the issues but this is a great and we will work on fixes/solutions.

On the other hand, I am just a bit curious about the use case. Would you mind sharing what the external monitoring application is doing with the rules?

Thank you,

Santiago.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/849d6a0e-be3b-454b-948c-bb5135f0d8a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Stoddard]

We are using the rules for a number of things. First, our company monitors very small companies - doctors offices, real estate offices, independent insurance companies, legal practices, CPA firms, not-for-profit associations and the like. These companies are an underserved market due to the cost of most tools today, and most of these companies do not have an IT staff - they use managed service providers (MSPs) to do their IT support work. Our platform is designed to work with MSP's to support these small clients, and because we have written our own software we can do this at a much lower cost than traditional SIEM solutions. Our solution is cloud based, so other than installing OSSEC and some other tools on their networks, there is very little IT knowledge required within these organizations to use our solution. MSP's that use our solution earn greater revenue by responding to alerts and correcting security issues, and clients get daily regulatory compliance reports showing them how well they are addressing compliance issues.

Regulatory compliance is the primary driver behind selling intrusion detection to small business. As a matter of fact, we released our service earlier this year (before we moved to OSSEC for event collection) and had to pull our product off the market a few weeks later because small firms don't think they are vulnerable to attack (it's a mindset that seems to pervade these firms). To say this took us by surprise is an understatement. But what these small businesses do need is regulatory compliance reporting, and they are willing to buy products that support that. The WAZUH branch of OSSEC handles PCI DSS compliance, but we needed more capability, and we needed to support additional regulatory standards - HIPAA, FINRA, GLBA, NIST 800-171, NIST 800-53, SOX, and ISO 27000+ to name a few. For example, we need to collect logs and archive them for clients. We also need to monitor for hosts, open ports, out of date software, and changes to client websites. And we need to track when things get modified, like accounts, logins, permissions, file and directory access, and the like. I doubt we will see much activity from SOX compliance as the companies that are required to do this are very large have their own IT staff to support expensive third party solutions, but out solution is flexible enough to meet almost any standard.

By pulling the rules out of OSSEC and into MySQL, we can map other regulatory compliance elements for other standards onto the OSSEC rule structure using tables in MySQL. As a matter of fact, this approach allows us to accommodate new regulations quickly, and we can update the OSSEC/WAZUH rules on our client sites without having to recode the groups back into the WAZUH XML rules to handle additional compliance regulations. We are doing a lot of work in this area, and we will spin off some of our work back to the community to help others in the development of their own SIEM solutions. Most of what we write is in Perl, and we have a very comprehensive API for our monitoring platform. We will open the API to the community sometime in the future.

We are a start up and we are in the process of raising investment capital right now for our launch, so I can't discuss a lot of details due to confidentiality constraints. However we are working with a lot of outside companies and building rules to support new hardware and software in OSSEC. We hope to be major contributors to the OSSEC open source platform in the months and years to come.  Best,

Dave Stoddard

Network Alarm Corporation

https://networkalarmcorp.com/

301-850-0668 x101

[Santiago Bassett]

Hi Dave, 

thanks for the detailed explanation of the use case. I do believe regulatory compliance is a huge driver for our users, so any contribution would be appreciated by the users community. 

We are also aware of this situation and working on a new release (planned for the end of September) that will greatly improve policy monitoring capabilities by making it possible to use SCAP content to check systems status and configuration. 

Best regards,

Santiago.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/398cf5a6-2f65-458d-baef-24c4ab932904%40googlegroups.com.