Utility function for testing panic

[snilsson]

It would be nice to have a utility for testing if a function panics
with a certain input. Would this submission to the testing package
have a chance of being accepted?

// Panics returns true if function f panics with parameters p.
func Panics(f interface{}, p ...interface{}) bool {
fv := reflect.NewValue(f)
ft := reflect.Typeof(f)

if ft.NumIn() != len(p) {
panic("wrong argument count")
}

pv := make([]reflect.Value, len(p))
for i, v := range p {
if reflect.Typeof(v) != ft.In(i) {
panic("wrong argument type")
}
pv[i] = reflect.NewValue(v)
}

return call(fv, pv)
}

func call(fv reflect.Value, pv []reflect.Value) (b bool) {
defer func() {
if err := recover(); err != nil {
b = true
}
}()

fv.Call(pv)
return
}

[snilsson]

It seems the enthusiasm for this kind of testing is less than
overwhelming, so let me share my experience.

1. Failing to raise a panic can be an insidious bug. The program
continues to run with a potentially broken internal state and may fail
much later with hard to debug issues.

2. Somewhat surprisingly, it's also a fairly common bug. This is the
way it tends to creep into code:

a) The original version of the code doesn't explicitly raise a panic;
it doesn't need to, since this clearly and obviously will happen
anyhow, e.g. by following a nil pointer. For crying out loud, it's
even in the documentation.

b) Later, you find a nice way to improve your code. This improvement
entails skipping that pointer... Whoops.

3. Even though the utility function is pretty straightforward, it's
nontrivial to implement for those of us who only use reflection
occasionally.

(By the way, the function probably should return the error value as
well.)

[Gustavo Niemeyer]

Hi there,

> It seems the enthusiasm for this kind of testing is less than
> overwhelming, so let me share my experience.

There's a strong preference for a minimalist design on testing, and to
be honest testing panics has been a bit rare in the standard library,
so that's why you're not getting much excitement.

That said, I do want to integrate something like this in gocheck:

http://labix.org/gocheck

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter

[snilsson]

On Apr 16, 4:30 pm, Gustavo Niemeyer <gust...@niemeyer.net> wrote:

> That said, I do want to integrate something like this in gocheck:
>
>    http://labix.org/gocheck

That would be nice!

Let me add another point to the list.

4. A test strategy that is likely to unearth nasty bugs is to check
for inconsistent internal state after a panic. Such code paths are
easy to miss during coding, rarely travelled during deployment, and
may result in delayed failure.

[Jan Mercl]

I think that idiomatic is to not allow panics get exposed at a package boundary and return errors instead. Thus package test suites don't have a need for such "expect-panic" test, IMO.

[David Roundy]

No, that's only true for some functions. The standard library does
have functions that panic on unexpected input. regexp.MustCompile is
an obvious example. I actually can't think offhand of any other
examples... but it's also not unreasonable to implement features
similar to slice and map accesses (which also panic) in one's own data
types.
--
David Roundy

[Jan Mercl]

2011-04-19 17:56:27 UTC+2 David Roundy wrote:

No, that's only true for some functions.  The standard library does
have functions that panic on unexpected input. regexp.MustCompile is
an obvious example.

It's an example of the other way around - intentionally turning a package API function returning error to a panic version. In this case testing the underlying function for properly returning the error on malformed arguments is IMO good enough.

http://golang.org/src/pkg/regexp/regexp.go?s=15018:15054#L626

 

 I actually can't think offhand of any other
examples...

There are some, e.g. this one will panic on illegal (out of bounds) index:

http://golang.org/src/pkg/container/vector/intvector.go?s=2339:2372#L86 

but it's also not unreasonable to implement features
similar to slice and map accesses (which also panic) in one's own data
types.

 Sure. My 2c opinion is just that it's probably not needed for the few/rare exceptions, which can be tested simply using defer w/o reverting to the reflection magic.

[nick....@gmail.com]

I try to write tests that use the least # lines of code, yet are fully explicit about their assumptions.

@Stefan big ups for testing panics. IMO writing software that fails consistently and verbosely is the #1 support for real-world sanity in ops.

I write a panic test like:

func TestAPI_Fail(t *testing.T) {

defer func(){

assert.NotNil(t, recover())

}()

DoSomething("that-should-panic")

}

Or for an explicit error:

func TestAPI_Fail(t *testing.T) {

defer func(){

err := recover()

assert.IsType(t, error, err)

assert.Equal(t, "Error message", err.(error).Error())

}()

DoSomething("that-should-panic")

}