Experimental HAProxy Log exporter

[Christopher Baines]

I am in the progress of writing an exporter for HAProxy logs [1]. Its
now working ok for me, so I'm publishing it in case anyone else is
interested in experimenting.

1: http://git.cbaines.net/prometheus-haproxy-log-exporter/

I'm aiming to export any information that is contained in the log, in a
configurable way (as it really needs configuring to match the
application you are using HAProxy for).

As a small example of the possibilities offered by this approach, I am
currently using it to get:

- session duration per the HTTP request path (this is only sensible if
you have a bounded small set of http_request_paths)

- request rates per backend, server and status code

- histograms for the server and backend queue lengths experienced by
each request

Let me know if you have any questions or comments. I'm particularly
interested in how best to configure the histogram buckets. I'm
considering trying to write a command that would read some log data and
then generate some sensible bucket configurations based on the observed
data (not sure how to do this yet though!).

[Brian Brazil]

On 23 February 2016 at 23:47, Christopher Baines <ma...@cbaines.net> wrote:

I am in the progress of writing an exporter for HAProxy logs [1]. Its
now working ok for me, so I'm publishing it in case anyone else is
interested in experimenting.

1: http://git.cbaines.net/prometheus-haproxy-log-exporter/

Have you looked at https://github.com/google/mtail which already has Prometheus support and can handle logs generally?

There's also https://github.com/prometheus/haproxy_exporter for the stats that haproxy itself exposes.

 

I'm aiming to export any information that is contained in the log, in a
configurable way (as it really needs configuring to match the
application you are using HAProxy for).

As a small example of the possibilities offered by this approach, I am
currently using it to get:

 - session duration per the HTTP request path (this is only sensible if
   you have a bounded small set of http_request_paths)

 - request rates per backend, server and status code

 - histograms for the server and backend queue lengths experienced by
   each request

Let me know if you have any questions or comments. I'm particularly
interested in how best to configure the histogram buckets. I'm
considering trying to write a command that would read some log data and
then generate some sensible bucket configurations based on the observed
data (not sure how to do this yet though!). 

You should pick your buckets in advance, otherwise you can't aggregate across servers.

--

Brian Brazil

www.robustperception.io

[Christopher Baines]

On 24/02/16 00:08, Brian Brazil wrote:
> On 23 February 2016 at 23:47, Christopher Baines <ma...@cbaines.net

> <mailto:ma...@cbaines.net>> wrote:
>
> I am in the progress of writing an exporter for HAProxy logs [1]. Its
> now working ok for me, so I'm publishing it in case anyone else is
> interested in experimenting.
>
> 1: http://git.cbaines.net/prometheus-haproxy-log-exporter/
>
>
> Have you looked at https://github.com/google/mtail which already has
> Prometheus support and can handle logs generally?

No I have not, I looked around for anything doing log processing, but
did not come across mtail. It looks interesting, although it does not
look possible to export histograms (yet).

> There's also https://github.com/prometheus/haproxy_exporter for the
> stats that haproxy itself exposes.

I am using that currently, and I think both exporters are very useful
and complementary. The haproxy exporter is very useful for gathering
overall stats, and the haproxy log exporter is very useful at gathering
metrics relating to individual requests.

> I'm aiming to export any information that is contained in the log, in a
> configurable way (as it really needs configuring to match the
> application you are using HAProxy for).
>
> As a small example of the possibilities offered by this approach, I am
> currently using it to get:
>
> - session duration per the HTTP request path (this is only sensible if
> you have a bounded small set of http_request_paths)
>
> - request rates per backend, server and status code
>
> - histograms for the server and backend queue lengths experienced by
> each request
>
> Let me know if you have any questions or comments. I'm particularly
> interested in how best to configure the histogram buckets. I'm
> considering trying to write a command that would read some log data and
> then generate some sensible bucket configurations based on the observed
> data (not sure how to do this yet though!).
>
>
> You should pick your buckets in advance, otherwise you can't aggregate
> across servers.

Good point, but I was thinking of this as more of a deployment utility
that would be a helpful too when setting up the exporter (not something
that runs automatically).

[Brian Brazil]

On 24 February 2016 at 00:20, Christopher Baines <ma...@cbaines.net> wrote:

On 24/02/16 00:08, Brian Brazil wrote:
> On 23 February 2016 at 23:47, Christopher Baines <ma...@cbaines.net
> <mailto:ma...@cbaines.net>> wrote:
>
>     I am in the progress of writing an exporter for HAProxy logs [1]. Its
>     now working ok for me, so I'm publishing it in case anyone else is
>     interested in experimenting.
>
>     1: http://git.cbaines.net/prometheus-haproxy-log-exporter/
>
>
> Have you looked at https://github.com/google/mtail which already has
> Prometheus support and can handle logs generally?

No I have not, I looked around for anything doing log processing, but
did not come across mtail. It looks interesting, although it does not
look possible to export histograms (yet).

> There's also https://github.com/prometheus/haproxy_exporter for the
> stats that haproxy itself exposes.

I am using that currently, and I think both exporters are very useful
and complementary. The haproxy exporter is very useful for gathering
overall stats, and the haproxy log exporter is very useful at gathering
metrics relating to individual requests.

If you're interested in individual requests, then you probably shouldn't be looking to get that data into Prometheus as it's a metrics store rather than a log store. Something like the ELK stack or graylog is likely more what you're looking for. See https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/

 

>     I'm aiming to export any information that is contained in the log, in a
>     configurable way (as it really needs configuring to match the
>     application you are using HAProxy for).
>
>     As a small example of the possibilities offered by this approach, I am
>     currently using it to get:
>
>      - session duration per the HTTP request path (this is only sensible if
>        you have a bounded small set of http_request_paths)
>
>      - request rates per backend, server and status code
>
>      - histograms for the server and backend queue lengths experienced by
>        each request
>
>     Let me know if you have any questions or comments. I'm particularly
>     interested in how best to configure the histogram buckets. I'm
>     considering trying to write a command that would read some log data and
>     then generate some sensible bucket configurations based on the observed
>     data (not sure how to do this yet though!).
>
>
> You should pick your buckets in advance, otherwise you can't aggregate
> across servers.

Good point, but I was thinking of this as more of a deployment utility
that would be a helpful too when setting up the exporter (not something
that runs automatically).

I'd generally adivse against this level of configurability for an exporter (https://docs.google.com/document/d/1JapuiRbp-XoyECgl2lPdxITrhm5IyCUq9iA_h6jp3OY/edit#heading=h.nsed3urztg8), particularly for histogram buckets where it's really important that everything you want to aggregate exports the same buckets. I don't think a good default would be hard to choose for this.

--

Brian Brazil

www.robustperception.io

[Christopher Baines]

On 24/02/16 00:28, Brian Brazil wrote:
> On 24 February 2016 at 00:20, Christopher Baines <ma...@cbaines.net

> <mailto:ma...@cbaines.net>> wrote:
>
> On 24/02/16 00:08, Brian Brazil wrote:
> > On 23 February 2016 at 23:47, Christopher Baines <ma...@cbaines.net <mailto:ma...@cbaines.net>

> > <mailto:ma...@cbaines.net <mailto:ma...@cbaines.net>>> wrote:
> >
> > I am in the progress of writing an exporter for HAProxy logs [1]. Its
> > now working ok for me, so I'm publishing it in case anyone else is
> > interested in experimenting.
> >
> > 1: http://git.cbaines.net/prometheus-haproxy-log-exporter/
> >
> >
> > Have you looked at https://github.com/google/mtail which already has
> > Prometheus support and can handle logs generally?
>
> No I have not, I looked around for anything doing log processing, but
> did not come across mtail. It looks interesting, although it does not
> look possible to export histograms (yet).
>
> > There's also https://github.com/prometheus/haproxy_exporter for the
> > stats that haproxy itself exposes.
>
> I am using that currently, and I think both exporters are very useful
> and complementary. The haproxy exporter is very useful for gathering
> overall stats, and the haproxy log exporter is very useful at gathering
> metrics relating to individual requests.
>
>
> If you're interested in individual requests, then you probably shouldn't
> be looking to get that data into Prometheus as it's a metrics store
> rather than a log store. Something like the ELK stack or graylog is
> likely more what you're looking for.
> See https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/

Maybe metrics relating to individual requests is not the right way of
putting it, but as per the example metrics I gave below, I am using
counters and histograms labeled with information contained in the log
lines (e.g. http_request_path, server_queue, ...) to build metrics (I am
not storing the log lines in Prometheus, just metrics).

I have made guesses at default bucket values [1], although now that you
mention it, it might be better to transform the unit of the data
(haproxy timers are in milliseconds, not seconds) rather than transform
the default bucket sizes (which I currently do).

1:
http://git.cbaines.net/prometheus-haproxy-log-exporter/tree/prometheus_haproxy_log_exporter/metrics.py#n79