Sort of misleading coverage stats

[Ondrej]

Hey!

There's an issue I've been grappling with for ages - say I have a custom function, Print, that calls six other functions, FormatString, FormatBool, FormatXYZ, ..., and I have no tests. Once I write some basic tests for Print, it does call some of the other functions and from looking at my test coverage, it looks like I have those nested functions covered, but I never called them explicitly, I never did try all their corner cases, did not fuzz them or anything.

It may sound odd or not important, but I have encountered this in almost every project in Go and other languages - the setup is always the same - lots of small functions doing their thing and then some orchestrator that integrates everything (could be a cli, could be an endpoint etc.) and once I set that up, coverage spikes up.

Here's a simple example:

// calculator.go

package calculator

func Add(a, b int) int {

return a+b

}

func Calculator(a, b int) int {

return Add(a, b)

}

// calculator_test.go

package calculator

import "testing"

func TestCalculator(t *testing.T) {

s := Calculator(1, 2)

_ = s

}

In this case, Calculator is covered, and so is Add, but I never tested how Add behaves for values close to the int maximum and with other potentially dangerous edge cases.

I've though about this over the years a tiny bit and never really found a great solution, because it's not really a technical thing, it's more about the workflow and attitude to testing.

I prototyped a simple tool that looks up all the exported functions and methods in a package and then checks if they are *explicitly* called in our tests. This does not guarantee anything, but at least we know they were explicitly used, not indirectly called within a different function. The prototype does not quite work for methods, because I got kind of lost in the go/{parser,ast} indirections. Also reassigning functions as first class values and then calling them from a different variable will probably also break it.

So it does give a lot of false positives for methods, but should be mostly fine for functions.

While it probably does not solve any problems for any people apart from me, I'll try and fix the method issues, because I do really need it for my own test writing.

Do let me know if there's a better solution to this or if you've encountered this and have some thoughts on the topic.

Thanks!

Ondrej

[Robert Engels]

It’s because you are not writing the top level test correctly - there is no testing at all !

On Mar 18, 2020, at 7:02 AM, Ondrej <ondrej...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/73963c74-bdde-4b22-8d59-cdce458e2877%40googlegroups.com.

<main.go>

[Ondrej]

That omission is not really relevant. Even if I included an assertion in the test, it wouldn't affect the point I am making. I just wanted to have at least some source code to reproduce the problem - adding if Calculator(1,2) != 3 {t.Error(...)} would make it more of a complete test, but my point is entirely different. My point is that top level functions calling further functions leads to "inflated" coverage numbers, because I am explicitly not testing any of the downstream functionality, just the top level function.

Again, this is not really a technical issue.

On Wednesday, 18 March 2020 13:29:05 UTC+1, Robert Engels wrote:

It’s because you are not writing the top level test correctly - there is no testing at all !

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[Robert Engels]

Exactly, but unless those functions are externally callable there is no reason to test them in isolation. 

On Mar 18, 2020, at 9:04 AM, Ondrej <ondrej...@gmail.com> wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f355f9d5-2a06-4f32-bf6c-f21fe61928d3%40googlegroups.com.

[Ondrej]

But they are callable - I'm only doing this for exported functions and methods - as I described in the initial post (and the implementation).

On Wednesday, 18 March 2020 15:15:30 UTC+1, Robert Engels wrote:

Exactly, but unless those functions are externally callable there is no reason to test them in isolation. 

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f355f9d5-2a06-4f32-bf6c-f21fe61928d3%40googlegroups.com.

[Robert Engels]

Im sorry then - I just assumed the tool would exclude or note the difference there. Personally code test coverage I take with a grain of salt because most people can’t write correct tests to begin with, so I’m almost always looking as tests in the low level packages that are used upstream. 

On Mar 18, 2020, at 10:12 AM, Ondrej <ondrej...@gmail.com> wrote:



To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f670f356-b7af-47bb-a3c0-76ee3df9b62e%40googlegroups.com.