questions about testing patterns and metric registration in Go

[Victor Watkins]

Hi! This question is about best practices for testing applications that use client_golang to instrument with custom metrics and avoiding the dreaded "duplicate metrics collector registration attempted".

Let's say I have an HTTP handler for an app that keeps a counter. The counter can be page hits, orders submitted, whatever. Let's say my handler, a struct with ServeHTTP attached, refers to this counter either as a package level variable or as a a struct member. I've seen the same behavior either way. Suppose I keep the typical patterns: use promauto, register my Collectors in New, or register my collectors at package initialization.

When I put my handler under test, I typically create a handler per test. This causes me to bump into "duplicate metrics collector registration attempted".

Some ways I have worked around this are to create a single httptest.Server with my handler in TestMain, or attach a RegisterMetrics to defer collector registration where I'm wiring up my dependencies (often main's main func) and just not register the metrics under test. The former makes it harder to inject mocks with different behaviors, the latter makes it hard to test my metrics.

So, the questions are:

1) What are some good ways to deal with metric registration under test?

2) Is testing metrics a typical practice, or do you just trust them? In the contrived example, it may be reasonably safe to just trust it, but I'm confident we have seen those cases where there is some tricky behavior around our metrics and we want to be sure we have it right.

3) When testing metrics, such as in an exporter concept where that is in fact the key behavior, is it typical to test the metrics endpoint with an http call (say, with httptest.Server) or do you look at the values held by the collectors?

Thanks!

Victor

[Brian Brazil]

On Sat, 4 Apr 2020 at 00:27, Victor Watkins <vwatk...@gmail.com> wrote:

Hi! This question is about best practices for testing applications that use client_golang to instrument with custom metrics and avoiding the dreaded "duplicate metrics collector registration attempted".

Let's say I have an HTTP handler for an app that keeps a counter. The counter can be page hits, orders submitted, whatever. Let's say my handler, a struct with ServeHTTP attached, refers to this counter either as a package level variable or as a a struct member. I've seen the same behavior either way. Suppose I keep the typical patterns: use promauto, register my Collectors in New, or register my collectors at package initialization.

When I put my handler under test, I typically create a handler per test. This causes me to bump into "duplicate metrics collector registration attempted".

That will only happen if you register in New, the other options will only ever register once. Accordingly you shouldn't register package-level variables in New.

Some ways I have worked around this are to create a single httptest.Server with my handler in TestMain, or attach a RegisterMetrics to defer collector registration where I'm wiring up my dependencies (often main's main func) and just not register the metrics under test. The former makes it harder to inject mocks with different behaviors, the latter makes it hard to test my metrics.

So, the questions are:

1) What are some good ways to deal with metric registration under test?

2) Is testing metrics a typical practice, or do you just trust them? In the contrived example, it may be reasonably safe to just trust it, but I'm confident we have seen those cases where there is some tricky behavior around our metrics and we want to be sure we have it right.

It depends, would you unittest a log line providing the same information? It's it's only a log line for debugging probably not, if it's a a first class feature for the given code (e.g. rpcs handled for a rpc library) then you should test it.

 

3) When testing metrics, such as in an exporter concept where that is in fact the key behavior, is it typical to test the metrics endpoint with an http call (say, with httptest.Server) or do you look at the values held by the collectors?

Whatever is easiest, going all the way to http sounds a bit much to me though.

--

Brian Brazil

www.robustperception.io

[Victor Watkins]

Thanks for sharing your wisdom and for correcting my misunderstanding about registering from init causing duplicate registration. I demonstrated both collectors as package level variables (package flipper) and struct members (package flopper) at https://github.com/vickleford/promex as an example.

I get your point about comparing a simple counter to a logger and not needing to put that under test. Thanks, that sanity-checks an opinion I had already formed. But suppose we're writing an exporter and that is our core logic that we want to prove under test. Following the above example for both styles, how do I read a metric without going as far as making an http call to /metrics? Perhaps https://github.com/vickleford/promex/blob/master/flopper/flopper.go#L68 asks this more succinctly than I can.

Looking forward to learning ^_^

[Bjoern Rabenstein]

On 04.04.20 12:44, Victor Watkins wrote:
>
> I get your point about comparing a simple counter to a logger and not needing
> to put that under test. Thanks, that sanity-checks an opinion I had already
> formed. But suppose we're writing an exporter and that is our core logic that
> we want to prove under test. Following the above example for both styles, how
> do I read a metric without going as far as making an http call to /metrics?
> Perhaps https://github.com/vickleford/promex/blob/master/flopper/flopper.go#L68
>  asks this more succinctly than I can.

You would usually call `Gather` on your registry and inspect the
result. https://pkg.go.dev/github.com/prometheus/client...@v1.5.0/prometheus/testutil?tab=doc
is a package to help you with that.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Victor Watkins]

Thanks, Bjoern! Does https://github.com/vickleford/promex/commit/bfa0880f56a480e2cb1f5ebac89e11e10bf6d938 capture your suggestion correctly? Am I unknowingly showing some anti-pattern that needs to get corrected? 

[Bjoern Rabenstein]

On 14.04.20 19:46, Victor Watkins wrote:
> Thanks, Bjoern! Does https://github.com/vickleford/promex/commit/

> bfa0880f56a480e2cb1f5ebac89e11e10bf6d938 capture your suggestion correctly? Am
> I unknowingly showing some anti-pattern that needs to get corrected? 

Yes, that's the way it is intended to be used.

In unrelated news, you might want to look at
https://prometheus.io/docs/practices/instrumentation/#failures ,
i.e. it might be better to include illegal flops in the flops_total
counter.