mtail + simple HAProxy config

[Panagiotis Atmatzidis]

Hello,

I'm trying to generate status code metrics from HAProxy logging system. The log file, configuration and debug output can be found on the respective gists. The problem is that I can't see any metrics.

The mtail config:

```
# metrics definition
gauge haproxy_active_connections
counter mtail_line_count by process, pid
counter http_requests_total by frontend_name_transport, backend_name, status_code, request_method, http_version
counter haproxy_http_nosrv_total by frontend_name_transport, backend_name, status_code, request_method, http_version
counter http_request_sent_bytes_total by frontend_name_transport, backend_name, status_code, request_method, http_version
counter http_request_backend_retries_total by backend_name, request_method

def syslog {
  /^(?P<date>(?P<legacy_date>\w+\s+\d+\s+\d+:\d+:\d+)|(?P<rfc3339_date>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+\d{2}:\d{2}))/ +
    /\s+(?P<hostname>[\w\.-]+)\s+(?P<application>[\w\.-]+)(?:\[(?P<pid>\d+)\])?:\s+(?P<message>.*)/ {
      len($legacy_date) > 0 {
        strptime($2, "Jan _2 15:04:05")
      }
      len($rfc3339_date) > 0 {
        strptime($rfc3339_date, "2006-01-02T15:04:05-07:00")
      }
      mtail_line_count[$application][$pid]++
      next
    }
}

@syslog {
  /\d+(\.\d+){3}:\d+ / +
  /(?P<frontend_name_transport>http-w+) / +
  /(?P<backend_name>\w+)\/<?(?P<server_name>[<>\w\.-]+)>? / +
  /(?P<status_code>\d{3}) / +
  /(?P<bytes_read>\d+) / +
  /(?P<client_query_time>-?\d+)\// +
  /(?P<actconn>\d+)\// +
  /(?P<feconn>\d+)\// +
  /(?P<beconn>\d+)\// +
  /(?P<srv_conn>\d+)\// +
  /(?P<retries>\d+) / +
  /(?P<srv_queue>\d+)\/(?P<backend_queue>\d+) / +
  /"(?P<request_method>[A-Z]+) (?P<URI>\S+) (?P<http_version>HTTP\/[0-9\.]+)"/ +
  /$/ {
    http_requests_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++
    /<NOSRV>/ {
      haproxy_http_nosrv_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++
    } else {
      http_request_sent_bytes_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version] += $bytes_read
      http_request_backend_retries_total[$backend_name][$request_method] += $retries
    }
    haproxy_active_connections = $actconn
  }
}
```

The `mtail_line_count` metric is visible, but the other metrics are not. I tested the regular expressions in regexp101 (golang). They match perfectly with the logs.

How can I debug further?

Kind regards,

P.

[Jamie Wilkinson]

Hi!  Thanks for using mtail.

The log file you provided has only 2 lines but the output example you gave has 3161 lines reported in the output.

But... you've got a `@syslog` decorator in your programme, and the line format doesn't look like it's prefixed with a syslog format message, so it's not matching the decorator before getting to your line match expression.

In your log line, what is the %ci:%cp representing?

I think you need to remove the syslog decorator and parse the haproxy timestamp directly.

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/91f0122d-eafe-4b5b-b1fe-0f16210ed877n%40googlegroups.com.

[Panagiotis Atmatzidis]

Hello!

Thanks for the reply.

You're correct. There are two errors in the above configuration. One is the decorator and another one is the log regex. Once these were fixed, everything worked as expected!

My configuration:

    counter mtail_line_count by process, pid

    counter mtail_haproxy_http_requests_total by frontend_name_transport, backend_name, status_code, request_method, http_version
    counter mtail_haproxy_http_nosrv_total by frontend_name_transport, backend_name, status_code, request_method, http_version

    histogram mtail_haproxy_http_response_duration_ms buckets 0.0, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0 by backend_name, server_name

    hidden gauge response_time_ms

    def syslog {
      /^(?P<date>(?P<legacy_date>\w+\s+\d+\s+\d+:\d+:\d+)|(?P<rfc3339_date>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\+\d{2}:\d{2}))/ +
        /\s+(?P<hostname>[\w\.-]+)\s+(?P<application>[\w\.-]+)(?:\[(?P<pid>\d+)\])?:\s+(?P<message>.*)/ {
          len($legacy_date) > 0 {
            strptime($2, "Jan _2 15:04:05")
          }
          len($rfc3339_date) > 0 {
            strptime($rfc3339_date, "2006-01-02T15:04:05-07:00")
          }
          mtail_line_count[$application][$pid]++
          next
        }
    }

    @syslog {
      /\d+(\.\d+){3}:\d+ / +

        /(?P<frontend_name_transport>http-\w+) / +

        /(?P<backend_name>\w+)\/<?(?P<server_name>[<>\w\.-]+)>? / +
        /(?P<status_code>\d{3}) / +
        /(?P<bytes_read>\d+) / +

        /(?P<response_time>-?\d+) / +
        /(?P<actconn>-?\d+)\// +
        /(?P<feconn>-?\d+)\// +
        /(?P<beconn>-?\d+)\// +

        /(?P<srv_conn>\d+)\// +
        /(?P<retries>\d+) / +
        /(?P<srv_queue>\d+)\/(?P<backend_queue>\d+) / +
        /"(?P<request_method>[A-Z]+) (?P<URI>\S+) (?P<http_version>HTTP\/[0-9\.]+)"/ +
        /$/ {

          # Handle response_time (%Tr):
          $response_time =~ /^-?\d+$|^0$/ {
            response_time_ms = 0.0
          } else {
            response_time_ms = $response_time / 1000.0
          }

          mtail_haproxy_http_requests_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++

          /<NOSRV>/ {
            mtail_haproxy_http_nosrv_total[$frontend_name_transport][$backend_name][$status_code][$request_method][$http_version]++
          } else {
            mtail_haproxy_http_response_duration_ms[$backend_name][$server_name] = response_time_ms
          }
        }
    }

The log format I'm using:

log-format "%ci:%cp %ft %b/%s %ST %B %Tr %ac/%fc/%bc/%sc/%rc %sq/%bq %%{+Q}r"

I'm sharing the configuration for someone else the might come across this problem. Debugging this took quite a bit, because when the regex doesn't match `mtail` will exit silently. So I started from scratch, line by line, working my way up using a `sample_metric` with a dummy value, checking if the metric was manifested or not.

Thank you & have a nice day!

[Jamie Wilkinson]

Glad it worked out.

If you have any ideas on how to make the debugging experience better please let me know.

To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/02dc8936-8cc4-426f-89a0-16a44f25c0c1n%40googlegroups.com.