Event time stamp in ElasticSearch.

[InfoSec]

Events in ElasticSearch are time stamped with the date the event was received by LogStash.

With the default behavior of the eventchannel collector picking up from where it left off if it is offline for any length of time, if the communication between OSSEC agent and server is disrupted, the timestamp of the event in ElasticSearch is off from the event generation time by at minimum the length of the disruption, plus the time it takes to clear the event backlog if lots of events have accumulated during the disruption.

This renders the all important event timestamp meaningless for investigation purposes. It also throws off any correlation engine that looks for a time-based sequence of events, unless it ignores the event timestamp and digs inside the event for the correct event date.

I have two questions:

1) How can the events be timestamped with the event generation time which is part of the event, rather than the time of reception by ElasticSearch?

A typical received json event looks as follows:

{"rule":{"level":1,"description":"Windows: <Event Description.>","id":XXXXX,"firedtimes":XXX,"groups":["windows","xxxxxx","xxxxxx"],"pci_dss":["XX.X.X"]},"agent":{"id":"XXX","name":"XXXXXXXX","ip":"XXX.XXX.XXX.XXX"},"manager":{"name":"xxxxxxx"},"dstuser":"XXXXXXX","full_log":"2017 Mar 14 09:55:23 WinEvtLog: <snip>,"timestamp":"2017-03-14T07:55:24+0000","location":"WinEvtLog"}

The timetstamp in Red is in the local timezone, the one in Blue is GMT.

2) How can the default behavior for eventchannel events be toggled to not go back where it left off when the agent went offline, but start picking up events from the log the moment the agent is started.

[Pedro Sanchez]

Hi,

Thanks for sharing your ideas and thoughts.

I think you are right about a few aspects, we had previously some talks and discussions about this topic, which keep on table for further updates or improvements.

What timestamp is being inserted/used for indexing at Elasticsearch?
We are using alerts field name "timestamp", which is generated by Wazuh when the event is processed (let's say, when it write the alert into alerts.json).

You could check it on the Logstash configuration: https://github.com/wazuh/wazuh/blob/master/extensions/logstash/01-wazuh.conf#L27

date {
    match => ["timestamp", "ISO8601"]
    target => "@timestamp"
}

Hey! You could use events timestamp, that will be really accurate.

Totally agree, but we would have some complications, for example:

- Not all the logs have timestamps

- The majority of them does not included timezone (which could be catastrophic, imagine inserting alerts on the future or 12 hours on the past) 

- Timestamp syntax is so different between logs sources, I mean, we could not guarantee the successful timestamp extraction or decoding.

- Currently Wazuh is not extracting timestamp as a field, we would love to do that, something like "real_timestamp" or "source_event_timestamp", but we would need to create pre-decoders or decoders to accomplish it.

EventChannel

I don't know a way to set up bookmarks behaviour on Wazuh, I know that by default it will start from the last saved bookmark:

https://github.com/wazuh/wazuh/blob/9edad706b47cd1e72bae138fb143545424a2911e/src/logcollector/read_win_event_channel.c#L400

"/* If we have a stored bookmark, start from it */"

Bookmarks are stored internally at C:\Program Files (x86)\ossec-agent\bookmarks folder, maybe a work around could be to remove them before start the agent (never tested).

Hope it helps, let me know if you did not understand something or I am not explaining myself properly.

I think this is a really interesting topic we will always love to discuss about.

Best regards,

Pedro 'snaow' Sanchez.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/eedbce1b-1ded-4d71-b8bd-b4cd91dc2236%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[InfoSec]

I have worked around the showstopper Windows events time stamp issue in logstash as follows:

Created a custom grok pattern in file /etc/logsash/patterns/custom-patterns with the following content:

WINEVTDATE \d\d\d\d \w\w\w \d\d \d\d:\d\d:\d\d

And added the following to /etc/logstash/conf.d/01-ossec-wazuh-filter.conf:

  grok {

    patterns_dir => [ "/etc/logstash/patterns/custom-patterns" ]

    match => [ "full_log", "%{WINEVTDATE:winevtdate}" ]

  }

  date {

    match => [ "winevtdate", "yyyy MMM dd HH:mm:ss" ]

    target => "@timestamp"

  }

This works for Windows events. In elasticsearch, the event timestamp for Windows events now matches the event generation time, but I presume only because in my setup elasticsearch, logstash, ossec server and ossec agent are all in the same time zone.

There may be gotchas. What would happen in the following cases:

1. What if there were multiple ossec servers each in a different timezone, and logstash/elasticsearch in yet another time zone?
   
2. What if the event in ossec was not a windows event, but included the date in a different format with no timezone information?
3. What if the event itself did not have a time stamp?

I think the solution to 1 & 2 is:

• If the event includes a date, have the OSSEC server use the time stamp in the received event as the OSSEC log time stamp.
• If time zone information is present along the date in the event, make use of it.
• If time zone information is not present along the date in the event, provide a mechanism to manually specify the time zone of the agent in the agent config on the server: /var/ossec/etc/shared/agent.conf?
  

For 3:

• Using the time that the ossec server receives and set the timezone to the server's timezone remain the only option.
  (I believe this is the currently adopted approach -- irrespective of cases 1 or 2, correct me if I am mistaken).

The ONLY time stamp that is usable for security correlations, investigations and reporting is the time the event was generated, with the assumption that the clock of the system that generated it is synchronized with a reliable reference (NTP). Given that every regulatory mandate explicitly insists on the integrity of the time stamp throughout the processing chain, Wazuh OSSEC fails dismally in this respect - it does work as designed, but the design is severely flawed! The information time stamp of events in elasticsearch can be meaningless, depending on how much time it took between generation time and reception time (highly variable, depending on how reliable/busy the infrastructure may be). With the OSSEC agent's default behavior of picking up events where it left off if there was any interruption in communication between itself and the ossec server, the setup as currently designed fails to satisfy - among many others - PCI requirement 10.4 and therefore "as is" cannot be utilized for PCI purposes.

I have managed to work around the limitation under very specific circumstances (this remains a limited workaround, THIS IS NOT A SOLUTION). The solution is to redesign the log output time stamp so under all circumstances it matches event generation time and time zone if they are present in the event.

On the second issue, stopping the aent, manually deleting the files under bookmarks, and restarting the agent works.

A configuration item should be added to the localfile section of the OSSEC windows agent config that toggles the behavior between the current default, and start capturing from the time the agent starts, ignoring anything that preceded. There is a need for both scenarios, depending on the circumstances we may want to proceed with the current behavior, or ignore what was not logged during an interruption for specific logs (especially if a huge backlog has built up), and start fresh.

Implementing an option as follows:

   <localfile>
      <location>Microsoft-Windows-DNS-Client/Operational</location>
      <log_format>eventchannel</log_format>

      <keep_bookmark>yes/no</keep_bookmark>
   </localfile>

"yes" will behave as is currently the case, "no" will delete the bookmark for the log file file in question upon agent start (or restart).

[Jesus Linares]

Hi George,

as Pedro mentioned, we have discussed this several times. OSSEC should be able to get the real date, as you said:

The ONLY time stamp that is usable for security correlations, investigations and reporting is the time the event was generated, with the assumption that the clock of the system that generated it is synchronized with a reliable reference (NTP).

Let's review the two difficulties that we find to solve this problem:

Extract the date

Every log has its own format. If we need the log date, what happens if:

1. The log doesn't have date.
2. OSSEC is not able to parse the date (what date should we use?)

This implies to have a pre-decoder phase that works with every type of log or improve every decoder to extract the date. This is the hardest part, because right now, it is not necessary any decoder to get the timestamp, so every kind of log will be analyzed by OSSEC without problems.

Deduce the timezone

Let's assume that we already have the date extracted. Also, let's assume that if the log contains the timezone we are able to extract it and use it. What should we do with the logs with no timezone?. We could use the agent timezone, but what happens if the agent is reading a log from rsyslog?. The remote server could have a different timezone. Should we leave the responsibility of correctly setting up rsyslog to display the timezone in the log to the user?.

We know that the current design is not perfect, but it works in every case because the timestamp is consistent for every log.

Change the timestamp is a delicated issue, for that reason we decided to use the date of the alert generation with the timezone of the manager. At least until we find a better solution. Elastic could be in any location because the alerts contains the date with the timezone.

Thanks for sharing your ideas. We will think about it. I hope that we find a good solution!.

Regards.