rsyslog file and alert in wazuh

[Nicolas Pasquarè]

Hello Guys,

i have configured rsyslog to read a log file, as is explained https://wazuh.com/blog/how-to-configure-rsyslog-client-to-send-events-to-wazuh/ here.

But, i cannot get the alerts in wazuh app on kibana. I have controlled and rules are matched correctly.

How i need to set in ossec.conf? or other things?

Thanks

[Mauro Ezequiel Moltrasio]

Hi Nicolas,

If wazuh is generating alerts on the received logs they should be forwarded to kibana automatically. You mention you checked the rules are matching correctly, can I ask how did you checked this? Did you use ossec-logtest or are alerts showing up in the alerts logs? Have you checked the ip on the server forwarding logs is included in the allowed-ips field of the remote configuration? You should also insure the manager is reachable from the server that is forwarding alerts with rsyslog, and if there are any firewalls you might need to enable access to the port.

Best,

Mauro

[Nicolas Pasquarè]

Hola Mauro,

buenas tardes, gracias por contestar.

De todas manera, cree un file con reglas nuevas, parecidas al del suricata, por que me llega un file json. 

La prueba la hice usando ossec-logtest -v, y me dio que hacia el match de las reglas justas.

Lo que no entiendo es como hacerle leer a Kibana el file de logs. Osea, yo cree un nuevo file sophos.conf en /etc/rsyslog.d/ 

$Modload imfile

$InputFileName /home/adminwazuh/sophos/log/central.json

$InputFileTag Sophos

$InputFileStateFile central

$InputFileSeverity alarm

$InputFileFacility local7

$InputRunFileMonitor

$InputFilePollInterval 30

El archivo "central.json" viene escrito cada tot tiempo por un batch, porque estamos usando esto https://github.com/sophos/Sophos-Central-SIEM-Integration

Lo que no logro hacer es hacerle leer las cosas nuevas.

Como puede hacer?

Gracias

[Mauro Ezequiel Moltrasio]

Hola Nicolas,

El primer paso sería asegurarnos que los mensajes que se están reenviando con rsyslog están llegando al manager de Wazuh. Para ello se puede utilizar la opción logall_json de la configuración global y corroborar que se están recibiendo los mensajes en los archives. Si no aparecen eventos asociados al servidor desde dónde se están reenviando los mensajes con rsyslogd, lo más probable es que haya un problema de conectividad entre los servidores, puedes intentar utilizar un puerto distinto al 514.

También veo que el servidor desde donde reenvías los logs tiene un usuario 'adminwazuh', si tienes un agente/manager en el mismo servidor puedes utilizar una configuración localfile en lugar de forwardearlo con rsyslogd.

Saludos,

Mauro.

[Nicolas Pasquarè]

Hola,

nosotros simplemente instalamos ese batch en el server Wazuh, que lo unico que hace es escribir los log en file central.json.

Lo que yo, todavia, no logre entender es como hacercelo leer al kibana. Porque al principio habia puesto dentro al ossec.conf esto:

<localfile>

  <log_format>json</log_format>

  <localtion>/directory/central.json</location>

</localfile>

Pero tampoco asi me andava, entonces, pense en hacer lo que te escribi arriba, osea hacerle leer el archivo al rsyslog que despues manda los datos a kibana. Pero nada, seguen vos puede ser que los datos nos lo escriban en alerts.json?

[Mauro Ezequiel Moltrasio]

Hola,

En este caso, yo volvería a colocar la configuración de localfile que tenías previamente y corroboraría que se están escribiendo alertas en el fichero alerts.json. Si el manager tiene bien configurado filebeat, las alertas deberían aparecer en kibana.

Saludos,

Mauro.

P.D.: Verifica también el nombre del fichero, son distintos entre la configuración de localfile y rsyslog que enviaste previamente ;)

[Nicolas Pasquarè]

Si si lo que escribi antes de local profile era para abreviar, pero si no seria "/home/adminwazuh/sophos/log/central.json"

Pensaba que lo que escribia nel ossec.conf era para los agent, no para el wazuh manager mismo.

Entoces, hago eso, y despues me tengo que fijar que los escriba en el alerts.json, si hay no estan, quiere decir que no me esta leyendo el file?

Y cada cuanto lo tiene controlado?

Gracias

[Mauro Ezequiel Moltrasio]

Hola Nicolas,

El localfile lo vas a tener que escribir en el agente que tiene acceso al fichero, una vez configurado y reiniciado, el agente va a empezar a hacer forward de los eventos que se escriban ahí. Si se están disparando alertas, tienen que aparecer en alerts.json, si no están, puedes configurar el manager para que genere un log de todos los eventos producidos con la opción logall_json en el tag global y corroborar en logs/archives/archives.json que se están recibiendo los mensajes del fichero.

Dependiendo de dónde aparezcan los mensajes puede pasar una de las siguientes condiciones:

 - Si no aparecen en el archives, algo de la configuración no está bien, se puede ver si hay algún mensaje en ossec.conf que nos indique qué está pasando.

 - Si aparecen en los archives pero no en alerts, hay que revisar las reglas con el campo full_log de los mensajes recibidos para entender qué tienen distinto.

 - Si aparece en los alerts pero no en kibana, hay que revisar logs y configuración de filebeat/elasticstack/kibana a ver dónde se está quedando.

Saludos,

Mauro.

[Nicolas Pasquarè]

Hola,

ahora veo las cosas en el alerts.json, pero no me aparecen en kibana:

{"timestamp":"2020-06-05T13:56:40.368+0200","rule":{"level":12,"description":"Sophos: Alert - HPmal/Eicar-A","id":"114502","firedtimes":1,"mail":true,"groups":["sophos-central"]},"agent":{"id":"000","name":"wazuhserver"},"manager":{"name":"wazuhserver"},"id":"1591358200.29644552","decoder":{"name":"json"},"data":{"id":"55efbe5c-5f76-409b-90f1-a534c3bb4f7d","rt":"2020-06-05T11:55:50.529Z","datastream":"alert","end":"2020-06-05T11:55:49.000Z","severity":"high","name": HPmal/Eicar-A","filePath":"C:\\Users\\xxx\\xxxx\\xxxx\\xxxx\\xxx\\sophos test\\sophos_hips_test.exe","type":"Event::Endpoint::Threat::HIPSCleanupFailed","detection_identity_name":"HPmal/Eicar-A","threat_cleanable":"true","event_service_event_id":"55efbe5c-5f76-409b-90f1-a534c3bb4f7d","dhost":"VMRTD43HQQ2","threat":"HPmal/Eicar-A","suser":"xxxx","customer_id":"26827de3-b979-49fc-a456-5555856dcb17","data":{"source_info":{"ip":"172.16.17.137"},"endpoint_id":"a6934a02-7feb-4cf0-bab7-057344e7be97","inserted_at":"1591358150528.000000","user_match_uuid":"de493234-4b39-0de7-445b-2158cc252497","created_at":"1591358150528.000000","endpoint_type":"computer","user_match_id":"5bd6d1fd8721671334042b6a","endpoint_java_id":"a6934a02-7feb-4cf0-bab7-057344e7be97","threat_id":"5d97135c3dd2870e5259da10","endpoint_platform":"windows","event_service_id":"55efbe5c-5f76-409b-90f1-a534c3bb4f7d"},"description":"Malware in esecuzione non rimosso: 'HPmal/Eicar-A' at 'C:\\Users\\xxx\\xxxx\\xxxx\\xxxx\\xxx\\\\sophos test\\sophos_hips_test.exe'"},"location":"/home/adminwazuh/sophos/log/central.json"}

Eso veo en el alert.json, y no entiendo donde y porque no lo veo en el kibana.

Filebeat tiene la configuracion :

filebeat.modules:

  - module: wazuh

    alerts:

      enabled: true

    archives:

      enabled: false

setup.template.json.enabled: true

setup.template.json.path: '/etc/filebeat/wazuh-template.json'

setup.template.json.name: 'wazuh'

setup.template.overwrite: true

setup.ilm.enabled: false

[Mauro Ezequiel Moltrasio]

Hi Nicolas,

Can you double check the alerts against a json validator? I took the one you pasted and there is a " missing in the data.name field, this could be a reason for filebeat/elastic not to take the alert into account. It might also be a good idea to look into filebeat/elasticsearch's logs for any error messages when forwarding the alert.

Best regards,

Mauro.

P.S.: Sorry for using english, but since this is a common place for users to come and find answers to problems it's a good idea to keep it neutral.

[Nicolas Pasquarè]

Hi Mauro,

it's not a problem.

Ok, but here is my ossec-logtest:

{"rt": "2020-06-05T11:55:50.529Z", "datastream": "alert", "end": "2020-06-05T11:55:49.000Z", "severity": "high", "name": "HPmal/Eicar-A", "filePath": "C:\\xxx\\sophos test\\sophos_hips_test.exe", "type": "Event::Endpoint::Threat::HIPSCleanupFailed", "detection_identity_name": "HPmal/Eicar-A", "threat_cleanable": true, "event_service_event_id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d", "dhost": "xxxx", "threat": "HPmal/Eicar-A", "suser": "VRX\\xxxx", "customer_id": "26827de3-b979-49fc-a456-5555856dcb17", "data": {"source_info": {"ip": "172.16.17.137"}, "endpoint_id": "a6934a02-7feb-4cf0-bab7-057344e7be97", "inserted_at": 1591358150528, "user_match_uuid": "de493234-4b39-0de7-445b-2158cc252497", "created_at": 1591358150528, "endpoint_type": "computer", "user_match_id": "5bd6d1fd8721671334042b6a", "endpoint_java_id": "a6934a02-7feb-4cf0-bab7-057344e7be97", "threat_id": "5d97135c3dd2870e5259da10", "endpoint_platform": "windows", "event_service_id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d"}, "id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d", "description": "Malware in esecuzione non rimosso: 'HPmal/Eicar-A' at 'C:\\xxx\\sophos test\\sophos_hips_test.exe'"}

**Phase 1: Completed pre-decoding.

       full event: '{"rt": "2020-06-05T11:55:50.529Z", "datastream": "alert", "end": "2020-06-05T11:55:49.000Z", "severity": "high", "name": "HPmal/Eicar-A", "filePath": "C:\\xxx\\sophos test\\sophos_hips_test.exe", "type": "Event::Endpoint::Threat::HIPSCleanupFailed", "detection_identity_name": "HPmal/Eicar-A", "threat_cleanable": true, "event_service_event_id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d", "dhost": "xxxx", "threat": "HPmal/Eicar-A", "suser": "VRX\\xxxx", "customer_id": "26827de3-b979-49fc-a456-5555856dcb17", "data": {"source_info": {"ip": "172.16.17.137"}, "endpoint_id": "a6934a02-7feb-4cf0-bab7-057344e7be97", "inserted_at": 1591358150528, "user_match_uuid": "de493234-4b39-0de7-445b-2158cc252497", "created_at": 1591358150528, "endpoint_type": "computer", "user_match_id": "5bd6d1fd8721671334042b6a", "endpoint_java_id": "a6934a02-7feb-4cf0-bab7-057344e7be97", "threat_id": "5d97135c3dd2870e5259da10", "endpoint_platform": "windows", "event_service_id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d"}, "id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d", "description": "Malware in esecuzione non rimosso: 'HPmal/Eicar-A' at 'C:\\xxx\\sophos test\\sophos_hips_test.exe'"}'

       timestamp: '(null)'

       hostname: 'wazuhserver'

       program_name: '(null)'

       log: '{"rt": "2020-06-05T11:55:50.529Z", "datastream": "alert", "end": "2020-06-05T11:55:49.000Z", "severity": "high", "name": "HPmal/Eicar-A", "filePath": "C:\\xxx\\sophos test\\sophos_hips_test.exe", "type": "Event::Endpoint::Threat::HIPSCleanupFailed", "detection_identity_name": "HPmal/Eicar-A", "threat_cleanable": true, "event_service_event_id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d", "dhost": "xxxx", "threat": "HPmal/Eicar-A", "suser": "VRX\\xxxx", "customer_id": "26827de3-b979-49fc-a456-5555856dcb17", "data": {"source_info": {"ip": "172.16.17.137"}, "endpoint_id": "a6934a02-7feb-4cf0-bab7-057344e7be97", "inserted_at": 1591358150528, "user_match_uuid": "de493234-4b39-0de7-445b-2158cc252497", "created_at": 1591358150528, "endpoint_type": "computer", "user_match_id": "5bd6d1fd8721671334042b6a", "endpoint_java_id": "a6934a02-7feb-4cf0-bab7-057344e7be97", "threat_id": "5d97135c3dd2870e5259da10", "endpoint_platform": "windows", "event_service_id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d"}, "id": "55efbe5c-5f76-409b-90f1-a534c3bb4f7d", "description": "Malware in esecuzione non rimosso: 'HPmal/Eicar-A' at 'C:\\xxx\\sophos test\\sophos_hips_test.exe'"}'

**Phase 2: Completed decoding.

       decoder: 'json'

       rt: '2020-06-05T11:55:50.529Z'

       datastream: 'alert'

       end: '2020-06-05T11:55:49.000Z'

       severity: 'high'

       name: 'HPmal/Eicar-A'

       filePath: 'C:\xxx\sophos test\sophos_hips_test.exe'

       type: 'Event::Endpoint::Threat::HIPSCleanupFailed'

       detection_identity_name: 'HPmal/Eicar-A'

       threat_cleanable: 'true'

       event_service_event_id: '55efbe5c-5f76-409b-90f1-a534c3bb4f7d'

       dhost: 'xxxx'

       threat: 'HPmal/Eicar-A'

       suser: 'VRX\xxxx'

       customer_id: '26827de3-b979-49fc-a456-5555856dcb17'

       data.source_info.ip: '172.16.17.137'

       data.endpoint_id: 'a6934a02-7feb-4cf0-bab7-057344e7be97'

       data.inserted_at: '1591358150528.000000'

       data.user_match_uuid: 'de493234-4b39-0de7-445b-2158cc252497'

       data.created_at: '1591358150528.000000'

       data.endpoint_type: 'computer'

       data.user_match_id: '5bd6d1fd8721671334042b6a'

       data.endpoint_java_id: 'a6934a02-7feb-4cf0-bab7-057344e7be97'

       data.threat_id: '5d97135c3dd2870e5259da10'

       data.endpoint_platform: 'windows'

       data.event_service_id: '55efbe5c-5f76-409b-90f1-a534c3bb4f7d'

       id: '55efbe5c-5f76-409b-90f1-a534c3bb4f7d'

       description: 'Malware in esecuzione non rimosso: 'HPmal/Eicar-A' at 'C:\xxx\sophos test\sophos_hips_test.exe''

**Phase 3: Completed filtering (rules).

       Rule id: '114502'

       Level: '12'

       Description: 'Sophos: Alert - HPmal/Eicar-A'

**Alert to be generated.

As you can see, it would be work correctly, the decoder works well. In filebeat o elasticsearch i don't find error. Log files are under /var/log? correctly?

Thanks

[Mauro Ezequiel Moltrasio]

Hi Nicolas,

I've tried your log on my system and it fails to add the alert to elasticsearch. The problem seems to arise from the log entry having a json field named 'data', after running through the default json decoder it gets translated into a field named data.data in the alert which our elastic template expects to be a string. The easiest solution would be to change that filed's name before it reaches elasticsearch, if you have access to the format the program is outputting to the log rename the field from data to programname-data or something similar. I think you can also achieve this using logstash but I don't have that much experience with it and might be a little overkill.

Best regards,

Mauro

[Nicolas Pasquarè]

Hi Mauro,

" if you have access to the format the program is outputting to the log rename the field from data to programname-data" this is impossible to me, because i have this from an API from cloud services, that in this case is Sophos.

Is there an other way?. Rename it with filebeat, or rename in an other way?

Thanks

[Mauro Ezequiel Moltrasio]

Hi Nicolas,

I believe you might be able to rename the field by using filebeat, but I have not been able to test it.

Take a look at the rename processor, you might be able to implement it into the default wazuh pipeline in order to rename the field in case the rule id matches.

https://www.elastic.co/guide/en/elasticsearch/reference/current/rename-processor.html

The Wazuh pipeline is located by default under the following location:

/usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json

Best regards,

Mauro Moltrasio.

[Nicolas Pasquarè]

Thanks for your feedbacks, about this unfortunaly i'm not in confidence with program so in deep.I have to study about.

[Nicolas Pasquarè]

Hello Mauro,

is there some of your collegues that maybe could help me?

Thanks

[Nicolas Pasquarè]

Hi Mauro,

how you troubleshoot this problem?.. is there a way for me to reproduce the same error?

Thanks

[Pablo Torres]

Hi Nicolas,

You can troubleshoot ingest failures as follows:

1. Edit the ingest pipeline, it is located in our Filebeat module: /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json 

You can see at the end of that file the field "on_failure", every time the ingestion fails that processor is triggered, by default you can see that we simply drop the event that failed so it won't be indexed in Elasticsearch, if you want to know the reason of that failure you just have handle the failure event: (you can find more info here: https://www.elastic.co/guide/en/elasticsearch/reference/master/handling-failure-in-pipelines.html)

The simplest way to see is adding a new field with the error, here's an example:

 "on_failure" : [
        {
            "set" : {
              "field" : "error",
              "value" : "{{ _ingest.on_failure_message }}"
            }
          }
   ]

Then you will have to apply the changes in the pipeline with this command:

filebeat setup --pipelines

That will add a new field, in this case it will be named error and it will contain the error message explaining why it couldn't be indexed.

The only difference is that failed alerts will be indexed in the filebeat index, in my case filebeat-7.7.1-2020.06.15 (that index name will depend on the current date and your filebeat version)

Now try indexing a new alert, and see if a new filebeat* index was created ( GET _cat/indices) then you will have to check what's inside that index (GET filebeat*/_search ) and this is what it should look like.

(To index a new alert you can just add it to the alerts.json file so Filebeat will think it was just generated by Wazuh, you can do that for testing purposes)

  "error" : """Unrecognized character escape 'x' (code 120)\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 1745]""",

You can see that it couldn't be indexed because of the filePath field: "filePath":"C:\xxx\sophos test\sophos_hips_test.exe"

to fix that, you will need to escape the character \

Those are the steps to troubleshoot ingest failures, if the ingest was correct but you still can't see the events in Kibana, there could be an error in your Elasticsearch, so you will have to check its logs:

cat /var/log/elasticsearch/(clusterName|elasticsearch).log

I hope it helps, let me know if I need to clarify any of the mentioned steps as it can be a little bit messy so I can give you further assistance!

Best Regards,

Pablo Torres

[pasquare...@gmail.com]

Hello Pablo,

thank you so much for you explication. 

i'm trying to reproduce what you tell me, but i can't found the new index "filebeat", so i can't find the error.

[pasquare...@gmail.com]

I obtain this from elastic:

2020-06-22T16:27:34,082][DEBUG][o.e.a.b.TransportShardBulkAction] [wazuh.local] [wazuh-alerts-3.x-2020.06.22][1] failed to execute bulk item (create) index {[<wazuh-alerts-3.x-{2020.06.22||/d{yyyy.MM.dd|UTC}}>][_doc][3f9t3HIBnTdRJN8yXOVB], source[{"agent":{"name":"wazuhserver","id":"000"},"manager":{"name":"wazuhserver"},"data":{"severity":"high","rt":"2020-06-22T14:27:20.175Z","data":{"endpoint_type":"computer","event_service_id":"01b8086c-7a10-42d2-8acd-4aa9f5c7911c","endpoint_platform":"windows","threat_status":"CLEANUPABLE","endpoint_id":"5a621fe3-5673-4468-a101-01a95b0b76eb","created_at":"1592836040171.000000","threat_id":"5ef0b1db88368f0e5c88a33c","user_match_uuid":"0a4e81e0-5234-907c-a33c-edb191490fa5","endpoint_java_id":"5a621fe3-5673-4468-a101-01a95b0b76eb","user_match_id":"5c580f2cd7b8ec131ba8440f","source_info":{"ip":"192.168.168.110"},"inserted_at":"1592836040171.000000"},"filePath":"C:\\zzzz\\xxxx\\folder\\sophos_hips_test.exe","description":"Malware in esecuzione non rimosso: 'HPmal/Eicar-A' at 'C:\\zzzz\\xxxx\\folder\\sophos_hips_test.exe'","type":"Event::Endpoint::Threat::HIPSCleanupFailed","threat_cleanable":"true","datastream":"alert","suser":"DOMAIN.LOCAL\\username","event_service_event_id":"01b8086c-7a10-42d2-8acd-4aa9f5c7911c","name":"HPmal/Eicar-A","end":"2020-06-22T14:27:18.000Z","id":"01b8086c-7a10-42d2-8acd-4aa9f5c7911c","threat":"HPmal/Eicar-A","detection_identity_name":"HPmal/Eicar-A","customer_id":"26827de3-b979-49fc-a456-5555856dcb17","dhost":"PCNAME"},"rule":{"firedtimes":1,"mail":true,"level":12,"description":"Sophos: Alert - HPmal/Eicar-A","groups":["sophos-central"],"id":"114502"},"decoder":{"name":"json"},"input":{"type":"log"},"@timestamp":"2020-06-22T14:27:31.899Z","location":"/home/adminwazuh/sophos/log/central.json","id":"1592836051.10171297","timestamp":"2020-06-22T16:27:31.899+0200"}]}

org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [data.data] of type [keyword] in document with id '3f9t3HIBnTdRJN8yXOVB'. Preview of field's value: '{endpoint_type=computer, event_service_id=01b8086c-7a10-42d2-8acd-4aa9f5c7911c, endpoint_platform=windows, threat_status=CLEANUPABLE, endpoint_id=5a621fe3-5673-4468-a101-01a95b0b76eb, created_at=1592836040171.000000, threat_id=5ef0b1db88368f0e5c88a33c, user_match_uuid=0a4e81e0-5234-907c-a33c-edb191490fa5, endpoint_java_id=5a621fe3-5673-4468-a101-01a95b0b76eb, user_match_id=5c580f2cd7b8ec131ba8440f, source_info={ip=192.168.168.110}, inserted_at=1592836040171.000000}'

[Pablo Torres]

Hi Nicolas!

Yes, the filebeat- index will only be created if there was an error in the pipeline, in this case you are receiving  a mapping exception:

...... org.elasticsearch.index.mapper.MapperParsingException:  failed to parse field [data.data] of type [keyword]

In our template, we specify the type of the field data.data as keyword, in your log this field comes with a JSON type so Elasticsearch throws a mapping error: 

"data":{"source_info":{"ip":"172.16.17.137"},"endpoint_id":"a6934a02-7feb-4cf0 .... }

You can fix it by renaming this field, you can do that editing our ingest pipeline (located here: /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json ):

More info about the rename processor can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/master/rename-processor.html

The new processor should look like this, you can see that I added a condition so it will only be renamed when the location is /home/adminwazuh/sophos/log/central.json (I took that value from the example log you shared in a previous reply, you can modify or remove it)

{

  "rename": {

    "if": "ctx.location == '/home/adminwazuh/sophos/log/central.json'",

    "field": "data.data",

    "target_field": "data_json",

    "ignore_missing": true,

    "ignore_failure": true

  }

},

To apply changes in the pipeline, you need to run this: 

filebeat setup --pipelines

Now your logs should be being indexed correctly, in this example the field data.data is renamed with the field data_json:

Please let me know if you have any other question!

Best Regards,

Pablo Torres