ANN: gocov, a Go coverage testing tool

[Andrew Wilkins]

Hi all,

I had a scratch to itch, thus I've been working on a coverage tool

for Go over the evenings of the past week. I'm aware there's the

broken "cov" tool, but for various reasons I didn't attempt to fix it.

Gocov is written entirely in Go, and instruments Go source code

to achieve coverage reporting.

The code is still pretty raw, and poorly documented, but I figured

I should let people know about it earlier rather than later, in the

event it helps someone.

Currently there's only support for simple statement coverage,

and only generating text reports. I'm happy to accept patches to

extend coverage to add condition coverage, otherwise I'll get to

it when I need it. Another obvious enhancement would be to add

HTML reports.

To install the gocov tool, "go get github.com/axw/gocov/gocov".

You can find some basic documentation on the usage at: https://github.com/axw/gocov

There is currently a limitation which means that gocov will only

be able to instrument and report on packages outside of

GOROOT, so you won't get statistics on standard library

packages. Below is an example of gocov's usage, for testing

"exp/types", which I have installed into my GOPATH:

$ gocov test exp/types > exptypes.json

... a bunch of output from "go test"

$ gocov report exptypes.json

...

types/exportdata.go      readGopackHeader              69.23% (9/13)

types/gcimporter.go      gcParser.expect                  66.67% (4/6)

types/gcimporter.go      gcParser.expectKeyword      66.67% (2/3)

...

$ gocov annotate exptypes.json types.gcParser.expectKeyword

266             func (p *gcParser) expectKeyword(keyword string) {

267                     lit := p.expect(scanner.Ident)

268                     if lit != keyword {

269 MISS                        p.errorf("expected keyword %s, got %q", keyword, lit)

270                     }

271             }

Feedback welcome. Enjoy.

Cheers,

Andrew

[Kyle Lemons]

Cool!  What's the performance like, compared to usual?  I have to admit that I've been quietly hoping that cov would be fixed for quite some time now.

On a side note:

https://github.com/axw/gocov/blob/master/gocov/main.go#L62

I think you mean to check for `key + "="` instead of "GOPATH="?

[Andrew Wilkins]

On Sun, Jul 8, 2012 at 4:12 AM, Kyle Lemons <kev...@google.com> wrote:

Cool!  What's the performance like, compared to usual?  I have to admit that I've been quietly hoping that cov would be fixed for quite some time now.

Well this of course depends on the number and type of statements, and I

haven't done extensive performance testing (read: nearly none), but in the

test of exp/types, there's a ~36% slowdown (0.926s / 0.679s).

 

On a side note:

https://github.com/axw/gocov/blob/master/gocov/main.go#L62

I think you mean to check for `key + "="` instead of "GOPATH="?

 

Derp. Thanks, Kyle, will fix.

Cheers,

Andrew

[Rodrigo Moraes]

On Jul 7, 5:12 pm, Kyle Lemons wrote:
> Cool!  What's the performance like, compared to usual?

Why do you ask? I mean, honest question. Is coverage expected to be a
ugly bottleneck?

-- rodrigo

[Kyle Lemons]

I was just curious.  If it slowed it down by an order of magnitude, I would think that the approach might not be broadly useful, but 30-50% (even 100%) slowdowns are reasonable.  To be honest, I very rarely care about coverage when writing my own code, but I do miss using coverage tools to double check when doing code reviews (i.e. when it's not a trivial matter to deliberately break a particular path and see if the corresponding unit test breaks).

[Paulo Pinto]

Depends on how it is done.

I have seen C applications require double the hardware to do coverage
analysis
when running, because of the runtime introspection happening.

--
Paulo

[Peter Kleiweg]

On 7 jul, 14:42, Andrew Wilkins <axw...@gmail.com> wrote:

> .... Below is an example of gocov's usage, for testing

> "exp/types", which I have installed into my GOPATH:
>
> $ gocov test exp/types > exptypes.json
> ... a bunch of output from "go test"

I tried this, and it doesn't work. What am I doing wrong?

(peter) ~/tmp godoc github.com/pebbe/go-proj-4/proj Proj
PACKAGE

package proj
import "github.com/pebbe/go-proj-4/proj"


TYPES

type Proj struct {
// contains filtered or unexported fields
}

func NewProj(definition string) (*Proj, error)

func (p *Proj) Close()


(peter) ~/tmp ~/go/bin/gocov test github.com/pebbe/go-proj-4/proj
failed to instrument package(.): no Go source files in /home/peter/
tmp
(peter) ~/tmp

[Andrew Wilkins]

On Sun, Jul 8, 2012 at 7:28 PM, Peter Kleiweg <pkle...@xs4all.nl> wrote:

    (peter) ~/tmp ~/go/bin/gocov test github.com/pebbe/go-proj-4/proj
    failed to instrument package(.): no Go source files in /home/peter/
tmp
    (peter) ~/tmp

I broke it after the announcement. Sorry about that, please try again, it should be fixed.

I tried to go get your project to test, but the repository's missing proj_api.h.

Cheers,

Andrew

[Peter Kleiweg]

On 8 jul, 15:37, Andrew Wilkins <axw...@gmail.com> wrote:

> On Sun, Jul 8, 2012 at 7:28 PM, Peter Kleiweg <pklei...@xs4all.nl> wrote:
>
> >     (peter) ~/tmp ~/go/bin/gocov test github.com/pebbe/go-proj-4/proj
> >     failed to instrument package(.): no Go source files in /home/peter/
> > tmp
> >     (peter) ~/tmp
>
> I broke it after the announcement. Sorry about that, please try again, it
> should be fixed.
>
> I tried to go get your project to test, but the repository's missing
> proj_api.h.

proj_api.h is part of PROJ.4. There is no pkg-config with PROJ.4 you
can use, so if it's installed in a non-standard location, you need to
set CPATH and LIBRARY_PATH for compiling and linking, and
LD_LIBRARY_PATH for running go-proj-4.

I get this with new version of gocov:

(peter) ~/tmp ~/go/bin/gocov test github.com/pebbe/go-proj-4/proj

panic: runtime error: invalid memory address or nil pointer
dereference
[signal 0xb code=0x1 addr=0x20 pc=0x405ea6]

goroutine 1 [running]:
main.(*instrumenter).instrumentPackage(0xf840057510,
0x7ffff44763bc, 0xf80000001f, 0x0, 0x0, ...)
/home/peter/go/src/github.com/axw/gocov/gocov/main.go:168
+0x34b
main.instrumentAndTest(0x7fff00000000, 0x7fff00000004)
/home/peter/go/src/github.com/axw/gocov/gocov/main.go:258
+0x5a5
main.main()
/home/peter/go/src/github.com/axw/gocov/gocov/main.go:313
+0x16e

goroutine 2 [syscall]:
created by runtime.main
/my/opt/go/src/pkg/runtime/proc.c:221
(peter) ~/tmp

[Andrew Wilkins]

On Sun, Jul 8, 2012 at 11:45 PM, Peter Kleiweg <pkle...@xs4all.nl> wrote:

proj_api.h is part of PROJ.4. There is no pkg-config with PROJ.4 you
can use, so if it's installed in a non-standard location, you need to
set CPATH and LIBRARY_PATH for compiling and linking, and
LD_LIBRARY_PATH for running go-proj-4.

Thanks, I can get it to build now.

 

I get this with new version of gocov:

    (peter) ~/tmp ~/go/bin/gocov test github.com/pebbe/go-proj-4/proj

    panic: runtime error: invalid memory address or nil pointer
dereference
    [signal 0xb code=0x1 addr=0x20 pc=0x405ea6]

I can reproduce this now - I missed the cgo files when parsing packages.

There's a few other things too, like retaining comments (for import "C")

I'll take a look at fixing this when I get home from work.

Cheers,

Andrew

[Andrew Wilkins]

On Mon, Jul 9, 2012 at 9:35 AM, Andrew Wilkins <axw...@gmail.com> wrote:

I can reproduce this now - I missed the cgo files when parsing packages.

There's a few other things too, like retaining comments (for import "C")

I'll take a look at fixing this when I get home from work.

Looks happier now.

andrew@fork:~/prog/gocov/gocov$ gocov test github.com/pebbe/go-proj-4/proj | gocov report

ok      github.com/pebbe/go-proj-4/proj 0.005s

proj/proj.go     Proj.Close      100.00% (3/3)

proj/proj.go     Transform2      100.00% (2/2)

proj/proj.go     DegToRad        100.00% (1/1)

proj/proj.go     NewProj         90.00% (9/10)

proj/proj.go     transform       66.67% (6/9)

proj/proj.go     Transform3      0.00% (0/1)

proj/proj.go     RadToDeg        0.00% (0/1) 

Cheers,

Andrew

[Peter Kleiweg]

On 9 jul, 16:27, Andrew Wilkins <axw...@gmail.com> wrote:

> Looks happier now.
>
> andrew@fork:~/prog/gocov/gocov$ gocov test
> github.com/pebbe/go-proj-4/proj| gocov report
> ok      github.com/pebbe/go-proj-4/proj 0.005s
>
> proj/proj.go     Proj.Close      100.00% (3/3)
> proj/proj.go     Transform2      100.00% (2/2)
> proj/proj.go     DegToRad        100.00% (1/1)
> proj/proj.go     NewProj         90.00% (9/10)
> proj/proj.go     transform       66.67% (6/9)
> proj/proj.go     Transform3      0.00% (0/1)
> proj/proj.go     RadToDeg        0.00% (0/1)

This a great. A very useful tool.

[kortschak]

I'm having some trouble with this now: it was working a few days ago.

I was testing biogo.llrb and previously this was fine, now it returns:
failed to parse gocov output: stat /tmp/gocov842622250/gocov.out: no such file or directory

after the tests complete (successfully).

It also fails on a new package (step) I'm working on which depends on llrb, but in this case it completes, warning that llrb is out of date (it isn't and go test does not give the same warning). It the proceeds to give me a detailed (though out of date) coverage report for llrb rather than a report for step.

thanks
Dan

[Andrew Wilkins]

On Tue, Jul 17, 2012 at 10:58 AM, kortschak <dan.ko...@adelaide.edu.au> wrote:

I'm having some trouble with this now: it was working a few days ago.

I was testing biogo.llrb and previously this was fine, now it returns:
failed to parse gocov output: stat /tmp/gocov842622250/gocov.out: no such file or directory

after the tests complete (successfully).

Is this on an uncommitted version of biogo.llrb? I just tested with the version on code.google.com/p/biogo.llrb, and gocov didn't have any troubles. Please note that I did update gocov a couple of evenings ago (it now supports instrumenting most packages in GOROOT). If you've updated gocov since then, perhaps I broke something.

Otherwise, if you could send me (off list) the code you're running through gocov, I'll take a look when I get home this evening.

 

It also fails on a new package (step) I'm working on which depends on llrb, but in this case it completes, warning that llrb is out of date (it isn't and go test does not give the same warning). It the proceeds to give me a detailed (though out of date) coverage report for llrb rather than a report for step.

I assume you mean the message "warning: building out-of-date packages"? It's because gocov creates a new, temporary GOPATH each time it runs, so for every run the package is in fact "out of date". This is as intended, ugly as it is.

If you can tell me how to reproduce the problem of "step" not being reported on, I'll see if I can diagnose.

Cheers,

Andrew

[kortschak]

The version of llrb I'm looking at is committed at googlecode.

The version of gocov I used when it was working was installed on my machine at home a few days ago and preceded the changes that allow cov on $GOROOT installed packages. The test today is on my work machine and was only installed about an hour ago, so I was thinking it might be that change.

I'll send you the step code off list (it's still in flux at the moment, so I haven't pushed it to googlecode yet), and what I did to fail.

thanks
Dan


On Tuesday, July 17, 2012 12:59:56 PM UTC+9:30, Andrew Wilkins wrote:

[Dave Cheney]

Hi Rob,

I'm not sure about that problem, it doesn't ring a bell. Did you log
an issue on https://github.com/axw/gocov/issues ?

On Mon, Jan 14, 2013 at 7:12 AM, <rob.k...@gmail.com> wrote:
> Is there any update on the "file not found problem?" I'm running into this
> on a newly installed copy of gocov on locally-developed code.
>
> Rob

> --
>
>

[Andrew Wilkins]

Hi Rob,

Echoing Dave: I'm not aware of any such problem currently existing.

If you could please log an issue on Github stating how to reproduce the problem, I will take a look.

Cheers,

Andrew

--

--
Andrew Wilkins

http://awilkins.id.au

[Dave Cheney]

The output from your failure would be a good place to start. 

On 14/01/2013, at 12:28, Rob Kinyon <rob.k...@gmail.com> wrote:

I will attempt to create a self-contained minimal test-case in the next few days.

One thing to note is that we use vagrant for our development environment. I don't know if that matters at all.

--
Thanks,
Rob Kinyon

[Rob Kinyon]

I will attempt to create a self-contained minimal test-case in the next few days.

One thing to note is that we use vagrant for our development environment. I don't know if that matters at all.

On Sun, Jan 13, 2013 at 7:08 PM, Andrew Wilkins <axw...@gmail.com> wrote:

--
Thanks,
Rob Kinyon

[Kamil Kisiel]

Mathias,

I was just using it today. Great stuff, and very useful :)

I noticed that sometimes it mis-indents blocks, but I can't seem to find any pattern to this behaviour. Have you noticed this at all?

On Saturday, January 26, 2013 4:18:08 PM UTC-8, matm wrote:

Hi,

just to let you know that I have published gocov-html, a simple tool for generating HTML reports from the output of gocov.

Documentation and source code can be found at https://github.com/matm/gocov-html.

Cheers,
Mathias

[Jan Mercl]

On Sun, Jan 27, 2013 at 4:58 AM, Kamil Kisiel <kamil....@gmail.com> wrote:
> I noticed that sometimes it mis-indents blocks, but I can't seem to find any
> pattern to this behaviour. Have you noticed this at all?

Is it https://github.com/matm/gocov-html/issues/1 ? If so, do you have
by chance still an old, pre fix, revision?

-j

[Jan Mercl]

On Sun, Jan 27, 2013 at 10:59 AM, Jan Mercl <0xj...@gmail.com> wrote:
> Is it https://github.com/matm/gocov-html/issues/1 ? If so, do you have
> by chance still an old, pre fix, revision?

Oh no, the latest revision actually has that bug reintroduced in just
a slightly different form. I've reopened issue #1 with a quick fix
(https://github.com/matm/gocov-html/issues/1#issuecomment-12753206).

HTH

-j

[Mathias Monnerville]

The code indentation bug is fixed (commit 1333b10de740e0e2f695446081faa05742d18e10).

Thanks for reporting!

--

Mathias

2013/1/27 Jan Mercl <0xj...@gmail.com>

[Kamil Kisiel]

Thanks for fixing it so quickly! 

[Greg Ward]

On 26 January 2013, matm said:
> just to let you know that I have published gocov-html, a simple tool for
> generating HTML reports from the output of gocov.

Nice! Thank you for that; it'll definitely make using gocov more fun.

Have you thought about turning gocov-html into a pull request against
gocov? It would be nice to have all that functionality in one
convenient place.

Greg
--
Greg Ward http://www.gerg.ca
<gr...@gerg.ca> @gergdotca

[Mathias Monnerville]

2013/1/31 Greg Ward <gr...@gerg.ca>

On 26 January 2013, matm said:
> just to let you know that I have published gocov-html, a simple tool for
> generating HTML reports from the output of gocov.

Nice! Thank you for that; it'll definitely make using gocov more fun.

Glad you like it!

Have you thought about turning gocov-html into a pull request against
gocov? It would be nice to have all that functionality in one
convenient place.

Well, I needed a quick way to get html formatting out of gocov (needed on a project), and I was aware that gocov's author felt that html output should be part of a separate package (https://github.com/axw/gocov/pull/10#issuecomment-8285228).

So, instead of forking it, I decided to create a separate package.

Best,
Mathias

[Andrew Wilkins]

Hi Oren,

If I'm following you correctly, I couldn't reproduce this. I did this:

 - created a package called "example" in GOPATH

 - go install example

 - gocov test example

... and it worked fine. Do you have the latest version of gocov? (go get -u github.com/axw/gocov/gocov)

If you've got the latest version of gocov and the problem persists, please file an issue on GitHub: https://github.com/axw/gocov/issues

Cheers,
Andrew

On Fri, May 24, 2013 at 3:55 PM, <or...@ben-kiki.org> wrote:

This is happening to me, and I pinpointed some of the "why", but I don't have a fix.

The problem seems to be that if there is an installed version of the being-tested package in my current workspace (which is the only entry in my GOPATH), somehow this version is run instead of the instrumented version in the newly created temporary workspace.

Looking at the source code I see that GOROOT is modified to point at the newly created temporary workspace, and looking at it (by using the --work flag) I see the instrumented package was successfully created there. And yet, the package is taken still taken from the original workspace - that is, from my GOPATH.

The error is, as one would expect:

ok      my/package/name   0.013s

failed to parse gocov output: stat /tmp/gocov072425766/gocov.out: no such file or directory

So, I added a line (next to setting the GOROOT) that sets GOPATH to be the newly created tempDir + ":" + the old value of GOPATH. I figured this will force finding the instrumented package, but this didn't make a difference(!?!?). Just for kicks I changed GOPATH to be only tempDir, and I got the expected warning that GOPATH has no effect, and of course running the tests failed due to missing dependencies - it did find the instrumented package itself though.

I am stumped at this point. My workaround is to simply remove the pkg directory before I run gocov, but that's a brutal workaround. Any ideas of a better solution would be welcome.

Thanks,

Oren Ben-Kiki

[Oren Ben-Kiki]

Yes, I am running the latest version (I just re-checked). And I agree, the code _should_ work, and doubly so with my force-the-GOPATH fix. But it doesn't :-( I realize that if this wasn't working for everyone else, you'd have noticed by now. Something must be different on my system...

If it matters, my disks are NFS-mounted on an IT-managed server that has a daemon running around taking snapshots and so on. I run into a problem with other tools when I discovered that last-modified-date of directories gets changed to the future(!) and back as part of that process. I have no idea if this is relevant here, though, I don't see that dates have anything to do with the problem.

I opened an issue #43.

Thanks,

Oren Ben-Kiki