What is the best exporter to monitor logs pattern

[Nabil LAABID]

Hi Folks,

I am currently looking for an exporter which will allow me to monitors somes logs (from WebLogic, Apaches or other application).

The aim is to find some specific pattern in the logs an raise an alert trough the Alertmanager.

Anyone can advise me which exporter is the best to perform this task (ideally an exporter that your are using)?

I've a look on the grok_exporter, but it seem a little bit hard to configure/handle...

Thanks a lot

[Brian Candler]

mtail is an alternative to grok_exporter.

There is also promtail, the frontend component of loki, which can expose prometheus metrics:

https://github.com/grafana/loki/blob/master/docs/clients/promtail/stages/metrics.md

Outside of the prometheus ecosystem, there are applications which are designed for matching log patterns and alerting on specific conditions - typically known as "HIDS" (host-based intrusion detection systems) - e.g. ossec/wazuh.  There are also tools which bolt onto the back of Elasticsearch - e.g. elastalert, or opendistro alerting.

[Ben Kochie]

Fluentd also the ability to do some metrics from logs.

https://github.com/fluent/fluent-plugin-prometheus#prometheus-outputfilter-plugin

Although, it doesn't seem nearly as powerful as mtail is.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a749697e-763b-4f6d-a1e0-c1c2bc6a10ba%40googlegroups.com.

[Daniel Trüssel]

On 04.03.20 18:26, Brian Candler wrote:
> Outside of the prometheus ecosystem, there are applications which are
> designed for matching log patterns and alerting on specific conditions
> - typically known as "HIDS" (host-based intrusion detection systems) -
> e.g. ossec/wazuh.  There are also tools which bolt onto the back of
> Elasticsearch - e.g. elastalert, or opendistro alerting.

there is also Splunk ($$$) in most enterprises

you can trigger an alert from Splunk to Prometheus with the REST Api scrape

[Cameron Kerr]

I would suggest that its useful to think of metrics as separate from logs/events. Similarly, I would treat exception handling (eg. Sentry) to be a separate leg of service monitoring, but that's beside the point.

Metrics have a rather more fixed cardinality and by design sacrifice detail, whereas logs tend to retain their richness (and often get further enriched).

Metrics are great for figuring out there are problems, but are quite limited in delving into why (eg. because your site is being crawled by a new type of spider), but you end up with a smaller mount of data.

Logs are great for figuring out why there are problems, and modern tools such as Elasticsearch make it possible to create performant dashboards even when you have mountains of data. You also tend not to worry about cardinality as much as you would with Prometheus

You can of course expose metrics within your log_processing, but from my experience, if you're using something like ElasticSearch, the aggregations and dashboards are quite performant, although there can be some overlap.

I've been using Elasticsearch for a few years now (Prometheus only for a few months long), and my most comprehensive dashboards are to do with web-sites. Before using Elasticsearch I was (and still am) very skilled at pulling apart logs using grep/sed/awk etc. but ELK (Elasticsearch, Logstash and Kibana) give me pretty much all the insight I've needed to be able to quickly drill down into a problems and help make data-informed improvements. I should also say, however, that this insight tends to come from a very verbose logging format I've specified in httpd (also nginx, which emits cleaner JSON) with a generous amount of enrichment from Logstash.

Indeed, you could even dispense with Elasticsearch and Kibana and just use Logstash for enriching logs and synthesizing metrics to be sent to... (well, typically statsd)

Coupled with this, I have used elastalert (albeit, not much) to look for things to alert about. According to the following, it can also integrate with alertmanager, although I haven't testing this myself yet.

https://github.com/Yelp/elastalert/pull/2228

Note also that Grafana can use Elasticsearch as a datasource too.

Hope that helps,

Cameron