Sending logs to Prometheus

[t12...@gmail.com]

Hi 

I am in the process of evaluating Prometheus and would like to know a few details regarding how best to analyze logs. Based on my research to convert logs to metric we have two options mtail or gork_export. how would i store all the logs in a centralise location for future analysis once an alert is triggered from Prometheus?

Do i need to have a separate Prometheus data base for storing metrics and another database like elastic search for storing the logs?

Thanks

[Ben Kochie]

Correct, Prometheus does not store logs or events, only metrics.

 

The design idea here is that Prometheus will alert you of problems and help narrow down which logs you need to look at, if necessary.

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f10ec892-d4cf-4da2-9c26-49619c5ae7f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[t12...@gmail.com]

So in a machine I have to install Gork_export to sent data to Prometheus and also file beat to send to elastic search? is this how most have implemented any suggestions would be greatly helpful. my only concert is that these tools might cause performance over head .

On Monday, May 8, 2017 at 11:07:23 AM UTC+5:30, Ben Kochie wrote:

On Mon, May 8, 2017 at 7:32 AM, <t12...@gmail.com> wrote:

Hi 

I am in the process of evaluating Prometheus and would like to know a few details regarding how best to analyze logs. Based on my research to convert logs to metric we have two options mtail or gork_export. how would i store all the logs in a centralise location for future analysis once an alert is triggered from Prometheus?

Do i need to have a separate Prometheus data base for storing metrics and another database like elastic search for storing the logs?

Correct, Prometheus does not store logs or events, only metrics.

 

The design idea here is that Prometheus will alert you of problems and help narrow down which logs you need to look at, if necessary.

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

On Mon, May 8, 2017 at 8:11 AM, <t12...@gmail.com> wrote:

So in a machine I have to install Gork_export to sent data to Prometheus and also file beat to send to elastic search? is this how most have implemented any suggestions would be greatly helpful. my only concert is that these tools might cause performance over head .

Yes, many people use both Prometheus and some kind of log tool.

That is one option.  There are many log shippers out there, beats, fluentd, logstash, etc.

However, the normal thing to do is to not use mtail/grok_exporter, but use a native client library.   These are much lighter weight when it comes to application overhead.  This is the main advantage of doing Prometheus pull-style metrics.  The overhead for doing metrics in-process are minimal, where as dealing with push messages and logs can cost a bit.

https://prometheus.io/docs/instrumenting/clientlibs/

 

On Monday, May 8, 2017 at 11:07:23 AM UTC+5:30, Ben Kochie wrote:

On Mon, May 8, 2017 at 7:32 AM, <t12...@gmail.com> wrote:

Hi 

I am in the process of evaluating Prometheus and would like to know a few details regarding how best to analyze logs. Based on my research to convert logs to metric we have two options mtail or gork_export. how would i store all the logs in a centralise location for future analysis once an alert is triggered from Prometheus?

Do i need to have a separate Prometheus data base for storing metrics and another database like elastic search for storing the logs?

Correct, Prometheus does not store logs or events, only metrics.

 

The design idea here is that Prometheus will alert you of problems and help narrow down which logs you need to look at, if necessary.

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f10ec892-d4cf-4da2-9c26-49619c5ae7f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/407d959e-f85d-44ae-8c4b-016273ee5fd1%40googlegroups.com.

[t12...@gmail.com]

Thanks for all the help and information. To start off this project I might not be able to get a buy in for instrumentation of application. Till such time is mtail/grok_exporter my only option or can any other alternatives you could suggest?

thanks

TK

[Ben Kochie]

No, those are the right options.  mtail/grok_exporter are good workarounds until you can directly instrument.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/488fc81a-b1cd-4af0-b154-1cc61c78d6f8%40googlegroups.com.