query regarding grok_exporter configuration

[Eva Sheeva]

Hello

I'm using grok exporter and here is what I want to achieve: I have a Java application whose log entry is in below format:

'%{NUMBER} %{JAVACLASS} %{JAVALOGMESSAGE} %{JAVATHREAD} %{TOMCAT_DATESTAMP} %{LOGLEVEL} %{JAVAMETHOD}'

I want to configure Prometheus alert for any 'ERROR' entry in the log level. Here is how the grok_exporter config.yml file look like:

====================

global:

    config_version: 2

input:

    type: file

    path: ./example/test.log

    readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines.

grok:

    patterns_dir: ./patterns

metrics:

    - type: counter

      name: error_test

      help: Counter metric example

      match: '%{NUMBER} %{JAVACLASS} %{JAVALOGMESSAGE} %{JAVATHREAD} %{TOMCAT_DATESTAMP} %{LOGLEVEL:severity} %{JAVAMETHOD}'

      labels:

            grok_field_name: severity

            prometheus_label: severity

server:

    host: 0.0.0.0

    port: 9144

============================ 

The test log file has 4 log lines with one log line having log level as ERROR. I did try  accessing http://IP:9144/metrics and I see the below but there is no metric created on Prometheus(grok_exporter is installed on Prometheus itself).

grok_exporter_line_processing_errors_total{metric="error_test"} 0
# HELP grok_exporter_lines_matching_total Number of lines matched for each metric. Note that one line can be matched by multiple metrics.
# TYPE grok_exporter_lines_matching_total counter
grok_exporter_lines_matching_total{metric="error_test"} 0
# HELP grok_exporter_lines_processing_time_microseconds_total Processing time in microseconds for each metric. Divide by grok_exporter_lines_matching_total to get the average processing time for one log line.
# TYPE grok_exporter_lines_processing_time_microseconds_total counter
grok_exporter_lines_processing_time_microseconds_total{metric="error_test"} 0
# HELP grok_exporter_lines_total Total number of log lines processed by grok_exporter.
# TYPE grok_exporter_lines_total counter
grok_exporter_lines_total{status="ignored"} 4
grok_exporter_lines_total{status="matched"} 0

Why is it so? What changes should I make to have the metric reflected on Prometheus? Can someone please help.

Thanks

[Eva Sheeva]

Can someone please help me on this
Thanks

[Eva Sheeva]

To add to, my logs are in json format as below:

{"@version":1,"source_host":"fstest-stage-bm-62","message":"Known host file not configured, using user known host file: /home/.ssh/known_hosts","thread_name":"Camel (camel-1) thread #4 - aws-s3://fstest-stage-bm-62","@timestamp":"2019-08-28T07:52:12.526+00:00","level":"INFO","logger_name":"org.apache.cam.file.remote.oerations"}

On Wednesday, September 11, 2019 at 6:38:02 PM UTC+5:30, Eva Sheeva wrote:

[Benoit Dubois]

These lines:

grok_exporter_lines_total{status="ignored"} 4
grok_exporter_lines_total{status="matched"} 0

Pretty much say your lines are analysed but the pattern is wrong. Your json log line has quotes and comas, your pattern don't take these into account.

[Eva Sheeva]

Hi Benoit

I tried using below format and i still couldn't get that to work. 

'{"%NUMBER":version},{"%JAVACLASS":source},{"%JAVALOGMESSAGE":message},{"%JAVATHREAD":threadname},{"%TOMCAT_DATESTAMP":timestamp},{"%LOGLEVEL":severity},{"%JAVAMETHOD":method}' 

Can you please help

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1faa4798-7989-445a-a18f-0c01a8d3fcec%40googlegroups.com.

[Benoit Dubois]

the quotation marks should be outside your patterns, and escaped. Also speaking of patterns could you paste them here? At first I tought you were combining key and value in your configs, but now I have a doubt. If you aren't combining them, you will also have to take into account the json keys.

'{%NUMBER:version}\",\"source_host\":\"{%JAVACLASS":source}\",\"message\":\"{%JAVALOGMESSAGE:message}\",\"thread_name\":\"{%JAVATHREAD:threadname}\",\"@timestamp\":\"{%TOMCAT_DATESTAMP:timestamp}\",\"level\":\"{%LOGLEVEL:severity}\",\"logger_name\":\"{%JAVAMETHOD":method}"

Be careful though, you might end up with very high cardinality in your metrics with a few of these patterns you're matching.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Eva Sheeva]

Hi Benoit, Thanks for your help. 

I tried usign the format that you mentioned, i do see the metric on Prometheus like earlier but there are no values seen. Below is the output:

-----------------------------------------------------------------------

# HELP grok_exporter_line_processing_errors_total Number of errors for each metric. If this is > 0 there is an error in the configuration file. Check grok_exporter's console output.
# TYPE grok_exporter_line_processing_errors_total counter
grok_exporter_line_processing_errors_total{metric="error_test"} 0
# HELP grok_exporter_lines_matching_total Number of lines matched for each metric. Note that one line can be matched by multiple metrics.
# TYPE grok_exporter_lines_matching_total counter
grok_exporter_lines_matching_total{metric="error_test"} 0
# HELP grok_exporter_lines_processing_time_microseconds_total Processing time in microseconds for each metric. Divide by grok_exporter_lines_matching_total to get the averge processing time for one log line.
# TYPE grok_exporter_lines_processing_time_microseconds_total counter
grok_exporter_lines_processing_time_microseconds_total{metric="error_test"} 0
# HELP grok_exporter_lines_total Total number of log lines processed by grok_exporter.
# TYPE grok_exporter_lines_total counter
grok_exporter_lines_total{status="ignored"} 3
grok_exporter_lines_total{status="matched"} 0
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} NaN
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} NaN
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="prometheus"} 0

------------------------------------------------------ 

 I have pasted the actual log format below. Just to clarify again, i want Prometheus to alert on "log level : ERROR" .

Kindly help me with this.

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/57a69f05-30f3-4464-979e-ca71e31c3cfe%40googlegroups.com.

[Benoit Dubois]

Are you using the exporter's default pattern definitions, or did you redefine them? I suspect a few of them like {%JAVATHREAD} and {%JAVAMETHOD} don't support special characters such #() and . in your strings if you are using the default ones.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/57a69f05-30f3-4464-979e-ca71e31c3cfe%40googlegroups.com.

[Eva Sheeva]

I'm using the default ones. I did not change anything. Can you please help how to do or point me to the documentation on how to do it. I tried to find a lot on this but i did not find much on this

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/847f4211-eee4-4b5f-9df6-68fc41388c77%40googlegroups.com.

[Benoit Dubois]

I'm not a regex expert but I can help out. Are you trying to capture thread name entirely or just "Camel (camel-1) thread #4"?

basically you have to define search patterns with an expressions such as
USERNAME [a-zA-Z0-9._-]+
You can alias a pattern by using it instead of declaring another regex like so:

USER %{USERNAME}

and you can join multiple patterns into a single one like this:

EMAILADDRESS %{USER}@%{HOSTNAME}

To help you out, basic regex notions can be found in this tutorial:
https://blog.usejournal.com/regular-expressions-a-complete-beginners-tutorial-c7327b9fd8eb

More info about grok patterns:
https://logz.io/blog/logstash-grok/

a list of the default patterns available to the exporter:

https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

and documentation about the regex engine:

https://github.com/kkos/oniguruma/blob/master/doc/RE

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/847f4211-eee4-4b5f-9df6-68fc41388c77%40googlegroups.com.