Custom Ingest Pipelines

[Matthew M.]

I'm trying to create an ingest pipeline that will allow me to utilize enrichment sources from Abuse.ch, IP-API, and Shodan to help populate data.

The problem I'm having with this is that I am not able to find ANYTHING relating to the creation of custom ingest pipelines in this manner. I know it's possible through the Opensearch and Wazuh documentation, but that's all I get is a reference to if it's possible and no where to start.

Is there any information available for this that the Wazuh dev team might be able to share?

[Roman Luna]

Hi,

You can see the current pipelines that we have in: /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json

I don't know about Abuse.ch, IP-API, and Shodan. Do you have an example of what you are trying to achieve?

Also, by default we only use FIlebeat for the pipelines which is limited to what they it does compared to, for example, Logstash. It should be taken into account to see if what you are looking for is possible or compatible with the services that you are using.

Regards.

[Matthew M.]

Roman,

For instance I'm trying to create a model for predicting when a Microsoft 365 account gets compromised.

What I'd like to do is feed the office365.clientIP field into a script that makes a call to IP-API which returns IP organizational data including whether it's a hosting or proxy address:

This would then sit inline with the event data and I can create rules around the proxy and/or hosting values.

I'm doing this right now with an integration script, but it's wildly inefficient. With an ingestion pipeline I could make it more efficient AND predictively add all IP fields into the pipeline.