I dont get it - "failed to parse [predecoder.timestamp]",

Corey H

unread,

Aug 10, 2018, 9:40:31 AM8/10/18

to Wazuh mailing list

Hi Everyone.

Over the last wile I keep getting a strange error in my logstash logs. This error is preventing windows agents from recording alerts in Elasticsearch.

Log Stash Log:

[2018-08-09T18:40:17,778][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2018.08.09", :_type=>"wazuh", :_routing=>nil}, #<LogStash::Event:0xbbab591>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2018.08.09", "_type"=>"wazuh", "_id"=>"XTLZIGUB06eDLlbFAm-K", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [predecoder.timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2018 Aug 09 18:38:58\" is malformed at \" Aug 09 18:38:58\""}}}}}

Versions

Wazuh Manager: 3.4.0

Filebeat and Logstash: 6.3.2

Elasticsearch 6.3.2

I am in Development, and strangely enough, it was working before I started working on enabling memory locking. (Not that this change should have anything to do with it)

Anyway I am useing the standard "remote" configurations for both filebeat and Logstash + username and password as I enabled the x-pack.

If is my understanding that this section (below) is what is reasonable for changing the date to a format that elasticsearch can understand

date {

match => ["timestamp", "ISO8601"]

target => "@timestamp"

}

But were is the predecoder.timestamp coming from... What should I change to make this work...

How should i update this config to make it work.

Thanks for your help "everyone"

Corey

Miguel Casares

unread,

Aug 10, 2018, 1:01:04 PM8/10/18

to Corey H, Wazuh mailing list

Hi Corey,

It is an unusual error, we need more information to help you.

Could you execute the following command in the Wazuh-Manager?

zcat /var/ossec/logs/alerts/2018/Aug/ossec-alerts-09.gz | grep "2018 Aug 09 18:38:58"

Could you also paste your Logstash configuration here please?

Best regards,

Miguel Casares

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b1c41207-1211-4a28-bc63-cce0429a19d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Miguel Casares

unread,

Aug 10, 2018, 1:06:36 PM8/10/18

to Corey H, Wazuh mailing list

Hi Corey,

I am sorry in my previous email I forgot to say something. When you execute the following command in the Wazuh-manager:

zcat /var/ossec/logs/alerts/2018/Aug/ossec-alerts-09.gz | grep "2018 Aug 09 18:38:58"

Please paste here the output too.

Best regards,

Miguel Casares

Corey H

unread,

Aug 10, 2018, 1:34:52 PM8/10/18

to Wazuh mailing list

Thanks so much for the reply....

I am recording the logs in txt and Json, so I have update the query to be

zcat /var/ossec/logs/alerts/2018/Aug/ossec-alerts-09.log.gz | grep "2018 Aug 09 18:38:58"

I got a lot back, but here is the end. (line in blue repeats) (please note I took out some hostnames etc)

2018 Aug 09 18:38:58 WinEvtLog: Security: AUDIT_FAILURE(4656): Microsoft-Windows-Security-Auditing: (no user): no domain: CA802172EPS01.Private.ShellAMRetail.com: A handle to an object was requested. Subject: Security ID: S-1-5-20 Account Name: HOSTA Account Domain: PRIVATE Logon ID: 0x3e4 Object: Object Server: PlugPlayManager Object Type: Security Object Name: PlugPlaySecurityObject Handle ID: 0x0 Process Information: Process ID: 0x2b4 Process Name: C:\Windows\System32\svchost.exe Access Request Information: Transaction ID: {00000000-0000-0000-0000-000000000000} Accesses: Unknown specific access (bit 1) Access Reasons: - Access Mask: 0x2 Privileges Used for Access Check: - Restricted SID Count: 0

2018 Aug 09 18:38:58 WinEvtLog: Security: AUDIT_FAILURE(4771): Microsoft-Windows-Security-Auditing: (no user): no domain: ShellAMDC02.Private.ShellAMRetail.com: Kerberos pre-authentication failed. Account Information: Security ID: S-1-5-21-2009767785-1111111111-1111111111-15718 Account Name: HOSTB Service Information: Service Name: SOME.DOMAN.LOCAL Network Information: Client Address: ::ffff:172.22.216.64 Client Port: 61043 Additional Information: Ticket Options: 0x40810010 Failure Code: 0x18 Pre-Authentication Type: 2 Certificate Information: Certificate Issuer Name: Certificate Serial Number: Certificate Thumbprint: Certificate information is only provided if a certificate was used for pre-authentication. Pre-authentication types, ticket options and failure codes are defined in RFC 4120. If the ticket was malformed or damaged during transit and could not be decrypted, then many fields in this event might not be present.

logstash config

---------------------------------------------------

[root@ShellAMKIB01 elasticsearch]# cat /etc/logstash/conf.d/01-wazuh.conf

# Wazuh - Logstash configuration file

## Remote Wazuh Manager - Filebeat input

input {

beats {

port => 5000

codec => "json_lines"

# ssl => true

# ssl_certificate => "/etc/logstash/logstash.crt"

# ssl_key => "/etc/logstash/logstash.key"

}

filter {

if [data][srcip] {

mutate {

add_field => [ "@src_ip", "%{[data][srcip]}" ]

}

if [data][aws][sourceIPAddress] {

mutate {

add_field => [ "@src_ip", "%{[data][aws][sourceIPAddress]}" ]

}

filter {

geoip {

source => "@src_ip"

target => "GeoLocation"

fields => ["city_name", "country_name", "region_name", "location"]

}

date {

match => ["timestamp", "ISO8601"]

target => "@timestamp"

}

mutate {

remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type", "@src_ip", "host"]

}

output {

elasticsearch {

hosts => ["localhost:9200"]

user => "logstash_internal"

password => "REMOVED"

index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"

document_type => "wazuh"

}

---------------------------------------------------------------

Here is some extras..

In prd (prof of concept) , I am doing everything the same with Wazuh 3.2 mins the addition of the cisco ASA logs.

For the asa, I have updated the wazuh-manager host syslog so save ASA logs to /var/log/remote_syslog.log

in the ossec.conf I have added this config to include the logs...

<log_format>syslog</log_format>

<location>/var/log/remote_syslog.log</location>

</localfile>

Looking at the JSON, I can see the winevtlog decoding as

predecoder":{"program_name":"WinEvtLog","timestamp":"2018 Aug 10 16:56:11"}

Looking at the JSON for the ASA it shows "predecoder":{"timestamp":"2018-08-10T11:57:07-04:00"}

The json for the ASA is getting into elasticsearh, but not the winevtlog...

Could this addition be hurting me in a Unpredictable manner.

Thanks Again..

Corey

Corey H

unread,

Aug 10, 2018, 1:57:08 PM8/10/18

to Wazuh mailing list

Again Thanks for the reply and I started to do some more testing....

I removed the ASA entry in the ocsec.config and deleted today index in elastic search.

restart the logstash and things started to get imported into elasticsearch from a WinEvtLog???

Could it be that the two decoders are messing me up?

Thanks Again!

Corey

Corey H

unread,

Aug 13, 2018, 10:36:54 AM8/13/18

to Wazuh mailing list

Hi Miguel,

I have done some testing over the weekend and it definitely related to me enabling the ASA (CISCO) logging in my syslog.

I configured my system to start local syslog messages to the normal /var/log/messages file and any remote messages to go to /var/log/remote_syslog.log

I then configured wazuh_manager to read the /var/log/remote_syslog.log so that i could disable the logging from the ASA.

Without recording of the remote syslog, and creating a new index I never have a issue.

If i enable the remote syslog and it is the FIRST log entry, I then get the error from above as it creates the index with the timestamp of "2018-08-10T11:57:07-04:00" and therefor "2018 Aug 10 16:56:11" is not valid

If i make sure that the index is created using the timestame ""2018 Aug 10 16:56:11" and then enable the remote system both are accepted.

An i doing something wrong? as I thought the config

date {

match => ["timestamp", "ISO8601"]

target => "@timestamp"

}

is supposed to be forcing me to the "2018 Aug 10 16:56:11" standard..

Can anyone help me with the config to deal with these two different date sources?

Corey

On Friday, 10 August 2018 07:40:31 UTC-6, Corey H wrote:

Corey H

unread,

Aug 14, 2018, 6:13:24 PM8/14/18

to Wazuh mailing list

Sorry, But I have to bump this one...

Bump

On Friday, 10 August 2018 07:40:31 UTC-6, Corey H wrote:

miguel....@wazuh.com

unread,

Aug 29, 2018, 1:40:45 PM8/29/18

to Wazuh mailing list

Hi Corey,

Sorry for the late response.

The timestamp format that appears in the windows JSON event you sent us is not the standard format for a JSON event. Could you tell me what version of Windows that agent is using?