Elastic ECS Compliance - or equivalent

[plop...@gmail.com]

Hi guys,

I have been using using Wazuh for 2-3 months now and I love it :). Thanks again for this work.

However, there is an important gap that I am not sure how to fix. I would like to get the log inserted into Elastic ECS compliance - or equivalent.

Indeed, I start getting numerous log into it and from numerous sources (Firewalls, IDS, Syslog, Proxy, etc.) so getting the logs following the same schema will be a huge benefits for correlating them with each other:

- Search across all the log types for the user "bla"

- Search across all the log types for the domain "blabla"

- Etc.

Right now, it's very difficult to do correlation searches or stuff a bit more advanced.

Si, if you don't mind I have multiple questions:

  1) Are you planning to do it? and if yes, would you know when?

  2) If it's not in your roadmap (I hope it is :P), what would be the best way to do it? In Wazuh or outside or it such as Logstash?

  3) If I do it in Wazuh, would there be some issues in the future that I might get from the tool itself? in future releases?

  4) If I do it outside, can I overwrite the fields by doing a rename directly in Logstash before insertion into ELK? or is it better to keep them and just do an alias to another name because Wazuh will need the original name somewhere? 

I will be glad if you could advice what is the best way to move forward ... I'm sure numerous people would be interested by that :)

Thank you,

Alex

[plop...@gmail.com]

Hello,

Do you have by chance some best practices? ideas?

Thanks,

Alex

[Kevin Branch]

Alex,

I agree that we need to move in the direction of some kind of normalized schema for at least field names that are common across diverse logging domains.  Pivoting is king in my book, and that's hard to do when there are a variety of ways we refer to things like source IP or destination port.  There are various standards out there like Elastic Common Schema (ECS), Splunk's Common Information Model (CIM), or ArcSight's Common Event Format (CEF), and I don't really know what would be best though I somewhat lean towards ECS right now.

How do others in the Wazuh community feel about using some kind of industry standard field naming convention to facilitate better pivoting and analytics on our log data?

Kevin

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/19fbe8a1-be3b-4fb2-b2af-1195ac421248%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rhys Evans]

I would like to see this, and think it would be a great addition

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[thebleucheese]

Agreed, this would be great, it'd make querying a lot easier and more compliant with other industry tools.

Conceptually it sounds a bit like it would be like Qradar's DSM mapper

[Kevin Branch]

Yes, just like the way that Qradar DSMs normalize extracted data to common field names after parsing.  Right now just perusing my Wazuh field names for Source IP address values, there are so many name variations depending on the log source:

data.srcip

data.src_ip

data.EventChannel.EventData.SourceIp

data.osquery.columns.src_ip

data.aws.sourceIPAddress

data.aws.source_ip_address

data.audit.srcip

This issue of diverse non-normalized field names is bigger now than before because Wazuh agents now send Windows logs in JSON format to the manager which means we see the Microsoft naming convention for all the fields, like instead of data.srcip we now see data.EventChannel.EventData.SourceIp from Windows agents.  I am very glad Wazuh enabled Windows logging via JSON because the parsing is so much more complete and reliable now, but it does have the side effect of much more diverse field names.  With Wazuh's native JSON log consumption supported not just for Windows, but for any JSON log sources, the diversity of field names will only continue to get worse unless we do some normalizing.

Field name normalization isn't a trivial issue to grapple with, either.  Do we rename fields from native to normalized names?  Do we copy the fields instead?  Would Elastic field aliases do the job?

While normalization is never perfect in the real world, it really makes a difference for quality analysis of log data, and it's an important feature in commercial SIEMs that I think we need to address in Wazuh to make our SIEM functionality even more compelling.

Kevin

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/15d530cb-d57a-499c-a4c3-25f135b13a77%40googlegroups.com.

[thebleucheese]

With field aliases wouldn't you need a distinct mapping for every type of log format you're collecting - https://www.elastic.co/guide/en/elasticsearch/reference/current/alias.html  would that be overly time-intensive to build, maintain, and update?

It looks like the way they're going with ECS, they're looking to capture the schema the same way in all of the beats which doesn't really help the effort here.  Maybe if there were logstash plugins to do the schema transform that would be helpful both for this project and the community as a whole looking to use ECS - especially for log ingestion that doesn't use beats.

I might be off here, I've only used elasticsearch in detail for searching large text sources with a pre-defined schema.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[plop...@gmail.com]

Thanks all for the your responses.

Yes, agree, normalization is the key. 

I have been using Splunk heavily in the past and it took us some time to set it up CIM compliance but once we successfully defined common fields for IP, username, user-agent, hostname, and plenty of others .. it was so much easier.

Now that I moved to ELK and Wazuh, I need to redo the job :P. 

Many vendors try to reinvent the wheel by creating another schema different from their competitor. To be honest, it doesn't really matter which one to use between the Splunk one, ELK, Qradar, etc. I'm just really interested in the fact that whatever the source log used, if I search for the field "src.ip", I get all of them. And it goes the same way for many common fields between log types.

The questions that I will be keen for the Wazuh team to reply are:

- Do you plan to do it? do you have a timeline?

- What is for you the best way to move forward between doing it straight into Wazuh or instead in Logstash?

- In Logstash, do you think it will be better to rename from native to normalized names ? copy the fields (don't think it is the best way to go for storage) or instead use aliases?

- Is there a possibility to break stuff for future updates?

Thanks,

Alex

[Pedro de Castro]

Hi team,

I am glad this matter came up in the community list. It is something we have been discussing internally since the beginning of the project.

- Do you plan to do it? do you have a timeline?

We currently do not have a timeline, but definitely, it is present on our daily discussions.

- What is for you the best way to move forward between doing it straight into Wazuh or instead in Logstash?

Below I am going to show what is our current idea, feel free to share what you think about it.

In my experience, everything we've done at Wazuh level delivers good results.

- In Logstash, do you think it will be better to rename from native to normalized names ? copy the fields (don't think it is the best way to go for storage) or instead use aliases?

The thing is, we have a Wazuh App in place which makes use of critical fields, if we rename those fields on Logstash layer, we can break Dashboards and others.

There is another consideration there, starting in Elastic 7.x, Wazuh does not longer require Logstash component (always welcome, but not mandatory).

- Is there a possibility to break stuff for future updates?

If we agree on how to do it and what is the standard, we will take care of deprecating the old naming in further versions.

Something to consider is if this effort must be released in a major version instead of major.minor.

Just to add here a little bit of historical background, on the early days OSSEC only support a set of eight fields which were able to extract data to, Wazuh added later the "Dynamic fields" which extends furthermore the field extraction and recently we have incorporated as well native JSON decoder which ingests JSON events right away.

The challenge now is to normalize all of those field names, we are not only thinking of the field names but as well improve the ruleset taxonomy (rule levels, rule groups and compliance mapping).

I agree with thebleucheese, Elastic field aliases cause too much overhead (in my experience), we were thinking more about adding aliases within Wazuh decoders syntax, an easy way to either add or rename the alert field name.

I'll illustrate here the concept we have in mind:

Aliases within raw log field extraction (decoding):

<decoder name="apache24-errorlog-ip-port">
    <parent>apache-errorlog</parent>
    <prematch offset="after_parent">[client \S+:\d+] \S+:</prematch>
    <regex offset="after_parent">[client (\S+):(\d+)] (\S+): </regex>
    <order>srcip,srcport,apache_id</order>
    <aliases>src_ip,src_port,id</aliases>
</decoder>
<decoder name="osquery-wcs-normalization">
  <parent>json</parent>
  <field action="rename">
    <source>data.osquery.columns.src_ip</source>
    <destination>src_ip</destination>
  </field>
  <field action="alias">
    <source>data.win.eventdata.subjectUserName</source>
    <destination>srcuser</destination>
  </field>
</decoder>

Wazuh can integrate as well with others indexers like Splunk and it is able to send data to other SIEMs, I am not sure if adapt our ruleset to ECS is the right call.

Will it make sense to have WCS (Wazuh Common Schema)? What do you think community?

Thanks again for the feedback and ideas, it is really exciting to see how the community grows, we will continue non-stop hard working to give you all the best open source security project on the planet.

[Rhys Evans]

Hi

Personally, I don't think there is any value coming up with a new "standard". Which standard to adopt is up for debate. 

But if you where looking at the aliasing type thing above, it would be possible to provide a mapping type file , that would allow you to "maintain" multiple standards?

Ie in config file you say I want my wazuh results in x standard

your field action would be something like if standard = x , destination is scr_user if standard =y , destination is srcuser

the issue here is the gui woudl need to read all standards

sorry just thinking out loud

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/15d530cb-d57a-499c-a4c3-25f135b13a77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/jJDrw0rz07g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6792e175-322e-43cc-abed-b03cdc7c35ee%40googlegroups.com.

[Dave Mound]

I'm also interested in this.  I'd like to start mapping OSSEC to ECS.  I've started creating mapping specs for other logs as I'm currently trying to standardise across our collect infra

Here is an example for Palo Alto - ECS https://docs.google.com/spreadsheets/d/1oNgWGNxA3HudUWUFImikC0gkyOpltwSpNWn5TwWmHOU/edit?usp=sharing

If this is something that would be of interest I would much appreciate a collab effort on mapping other sources 

Regards,

Dave

[Kevin Branch]

Pedro, you have a point about Logstash not being for everyone now, so perhaps focusing on some kind of common schema mapping would best be done at the filebeat or template or ingest processor level.  I read there is a copy_to option you can use in templates to copy a field, but it only copies the value, not the tokens/terms resulting from analysis of the field to copy.  I see an option in Filebeat to rename fields but I'm not sure there is a way at that level to copy a field.  It appears that the Elasticsearch ingest node "dissect" processor might be usable to copy fields, but I'm not sure if it would do so before or after field analysis.

As to copying vs renaming fields for common schema purposes, though it costs more storage I'd still vote for copying.  That would better keep the Wazuh Kibana app compatible with its native field names.  Otherwise changes in an external standard like ECS might jerk us around a bit depending on how stable it proves to be.  Copying would let us enjoy the advantages of a schema standard that is bigger than Wazuh, allowing us to use additional shared tools and dashboards out there that would be ECS-compliant, without sacrificing our compatibility with Wazuh native field names.  Best of both worlds :)  

Kevin

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/15d530cb-d57a-499c-a4c3-25f135b13a77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6792e175-322e-43cc-abed-b03cdc7c35ee%40googlegroups.com.