Proposed new pkg/testing function

690 views
Skip to first unread message

Steven Degutis

unread,
Feb 15, 2012, 7:28:43 PM2/15/12
to golang-nuts
The 'testing' package is great! But there's one thing about it that
keeps making it difficult to use.

Currently, when I use Errorf (etc), I see the file and line printed
out. That's very helpful. But when I try to write a helper function
for myself which internally uses Errorf, I see the file and line of my
helper function's invocation of Errorf printed out.

I'm not suggesting making a major change to the testing package.
According to the Open-Closed Principle (http://www.objectmentor.com/
resources/articles/ocp.pdf) a class (or in this case package) should
be open to extension but closed to change. The Go team's philosophy
seems very much in harmony with this idea, especially in keeping
things backwards compatible.

The Go team also seems intent on packages being minimalist and not
adding use-case-specific features where they don't belong. That's why
I propose we shouldn't add too much to the 'testing' package, but
leave it mostly identical to what it is now, only adding a single
function to assist with extension.

Recently, I've wanted to write an AssertEquals() family of functions.
This should rightly be third party, living outside of the 'testing'
package. But it should be able to leverage the 'testing' package.

So the change I'm proposing is one new and isolated function in
'testing':

func (c *T) IndirectlyLogf(format string, args ...interface{})

This function would be identical to Logf() except it would print out
the file:line one higher in the stack trace than its existing
counterpart. This function would be useful for people like me to
experiment with new testing helper functions that directly use the
'testing' package.

It would be an isolated function, not touching or breaking anything
else. It wouldn't be a new feature in itself, but rather an extension
that allows for new features.

To give an example use-case where I would like to see it, this is a
function I have written with it in mind:

func AssertEquals(t *testing.T, a interface{}, b interface{}) {
if a != b {
t.IndirectlyLogf("expected %v to equal %v", a, b)
t.Fail()
}
}

Now this may not be the best or ideal use-case for IndirectlyLogf(),
but the point is to allow experimentation so even if my idea sucks,
someone else can try something else, and the community can build upon
each other's ideas.

It will be very easy to implement this function. I volunteer to do it
myself. In fact I wrote it just now, right before finishing this
sentence. It was super easy. All we needed to do was add one more bool
parameter to decorate() which dictates whether Caller(3) should be
Caller(4) or not, and this function would use that version of
decorate.

-Steven

Jan Mercl

unread,
Feb 16, 2012, 3:09:55 AM2/16/12
to golan...@googlegroups.com
On Thursday, February 16, 2012 1:28:43 AM UTC+1, Steven Degutis wrote:

To give an example use-case where I would like to see it, this is a
function I have written with it in mind:

        func AssertEquals(t *testing.T, a interface{}, b interface{}) {
                if a != b {
                        t.IndirectlyLogf("expected %v to equal %v", a, b)
                        t.Fail()
                }
        }

I don't like the idea.

        // This is plain/simple and customizable for every equality test
        if g != e {
                t.Fatalf("foo %% bar: got %v exp %v in pass %d, %d is not prime", g, e, i, p)
        } 

        // vs
        AssertEqual(t, g, e)

Paul Borman

unread,
Feb 16, 2012, 12:50:02 PM2/16/12
to Jan Mercl, golan...@googlegroups.com
While I agree with Jan about this particular example, I think the point that Steven is making is missed.  There are situations where tests repeatedly do the same multi-line checks (testing err is a good example).  It is much better to write a function to do this test but doing so causes the problem that Steven points out.

There is a hacky solution:

func testErr(t *testing.T, err error, s string) {
        _, _, line, _ := runtime.Caller(1)
        switch {
        case err == nil && s == "":
                return
        case err == nil:
                t.Errorf("%d: Expected error %q, got none", line, s)
        case s == "":
                t.Errorf("%d: Unexpected error %q", line, err)
        case !strings.Contains(err.Error(), s):
                t.Errorf("%d: Got error %q, expected %q", line, err, s)
        }
}

You end up having two line numbers printed.  I would find it nice to be able to set the location of where I want the error to appear.  Off the top of my head something like:

// SetCaller fixes reporting location to be
// level frames above the current frame.
// The function returned reverts the reporting
// location to the previous value.
//
// Example:
//
//    defer t.SetCaller(1)()
func (t *T)SetCaller(level int) (func ())

where t.SetCaller sets t

Steven Degutis

unread,
Feb 16, 2012, 1:40:19 PM2/16/12
to golang-nuts
The current solution of the 'testing' package seems temporary, so when
I proposed a solution, I didn't propose a completely ideal one, but
rather one that allows us to keep using 'testing'. This change gives
us relatively a lot of extra flexibility, with relatively little
addition to the stdlib code, and it's completely isolated from the
rest of the code.

In the future though, I think we could introduce a more flexible
testing package that uses interfaces instead for extensibility. But
that's not pertinent to this proposed code addition, which will keep
us going with good old 'pkg/testing' for quite a while I would think.

-Steven
> >>                         t.**IndirectlyLogf("expected %v to equal %v", a,
> >> b)
> >>                         t.**Fail()

Andrew Gerrand

unread,
Feb 16, 2012, 6:17:27 PM2/16/12
to Steven Degutis, golang-nuts
Helper functions like AssertEquals are antithetical to the testing
package's philosophy.

AssertEquals provides no information, just a line number and whether
the assertion was true or false. This call

AssertEquals(t, foo, bar)

can, at most, say something like "At line x: fooval != barval". Add a
new arg to give context:

AssertEquals(t, "Quux(args)", foo, bar)

so now you have "at line x: Quux(args) == fooval, want barval" which
is a somewhat informative if rigidly formatted error message. And you
need to remember the order of arguments for your AssertEquals
function. This kind of design quickly spirals into dozens of AssertFoo
variations which aren't much fun to use.

Give me this any day:

if foo != bar {
t.Errorf("Quux(%q) == %q, want %q", args, foo, bar)
}

Unless you can propose a more palatable use case for IndirectlyLogf, I
don't think you'll get much traction with this idea in the core Go
testing package.

Andrew

Jamu Kakar

unread,
Feb 16, 2012, 7:04:42 PM2/16/12
to Steven Degutis, golang-nuts
Hi Steven,

On Thu, Feb 16, 2012 at 4:40 PM, Steven Degutis <sdeg...@8thlight.com> wrote:
> The current solution of the 'testing' package seems temporary, so when
> I proposed a solution, I didn't propose a completely ideal one, but
> rather one that allows us to keep using 'testing'. This change gives
> us relatively a lot of extra flexibility, with relatively little
> addition to the stdlib code, and it's completely isolated from the
> rest of the code.
>
> In the future though, I think we could introduce a more flexible
> testing package that uses interfaces instead for extensibility. But
> that's not pertinent to this proposed code addition, which will keep
> us going with good old 'pkg/testing' for quite a while I would think.

You might consider gocheck, which is more full-featured by design and
built atop the testing package:

http://labix.org/gocheck

It has matchers which let you extend it in interesting and novel ways
and comes with a set of basic things, so you can write code like:

c.Assert(foo, Equals, bar)
c.Assert(err, IsNil)

... and so on. Right now the only issue I have with it is that 'go
test' doesn't accept gocheck specific command-line arguments (at
least, not in golang-weekly 2012-02-07-1~11764~oneiric1 on Ubuntu),
but I guess that's a temporary one.

Thanks,
J.

Gustavo Niemeyer

unread,
Feb 16, 2012, 7:43:38 PM2/16/12
to Jamu Kakar, Steven Degutis, golang-nuts
Hey man,

> ... and so on.  Right now the only issue I have with it is that 'go
> test' doesn't accept gocheck specific command-line arguments (at
> least, not in golang-weekly 2012-02-07-1~11764~oneiric1 on Ubuntu),
> but I guess that's a temporary one.

I don't think the command line arguments ever stopped working. What's
the issue you're facing there?

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/plus
http://niemeyer.net/twitter
http://niemeyer.net/blog

-- I'm not absolutely sure of anything.

Kyle Lemons

unread,
Feb 16, 2012, 8:43:27 PM2/16/12
to Andrew Gerrand, Steven Degutis, golang-nuts
I mostly tune out the line numbers because, in table driven tests, they only tell me which test failed and not which test case failed, which I put in the message anyway.  I'd like a t.SetFlags(0) or something ala log.SetFlags so that I could configure what is printed for the test.

Gustavo Niemeyer

unread,
Feb 16, 2012, 9:25:23 PM2/16/12
to Andrew Gerrand, Steven Degutis, golang-nuts
Hey Andrew,

> AssertEquals provides no information, just a line number and whether
> the assertion was true or false. This call
>
>  AssertEquals(t, foo, bar)
>
> can, at most, say something like "At line x: fooval != barval". Add a

This is somewhat of a pessimistic view on what one can do.

Given this:

func Quux(s string) string { return "baz" }

func (S) TestDetails(c *C) {
c.Assert(Quux("foo"), Equals, "bar")
}

This is the real output from gocheck

----------------------------------------------------------------------
FAIL: foo_test.go:21: S.TestDetails

foo_test.go:22:
c.Assert(Quux("foo"), Equals, "bar")
... obtained string = "baz"
... expected string = "bar"

liigo

unread,
Feb 16, 2012, 10:04:30 PM2/16/12
to golang-nuts
+1 for AssertEqual(). It's much easier to use (less code to type) and
to understand, it has been widely used in other languages / test-
frameworks, such as JUnit and TestNG. I would like to see AsserEqual()
adding to testing pkg, please. Thanks.

Cheers,
Liigo Zhuang

Gustavo Niemeyer

unread,
Feb 16, 2012, 10:06:56 PM2/16/12
to liigo, golang-nuts
On Fri, Feb 17, 2012 at 01:04, liigo <com....@gmail.com> wrote:
> +1 for AssertEqual(). It's much easier to use (less code to type) and
> to understand, it has been widely used in other languages / test-
> frameworks, such as JUnit and TestNG. I would like to see AsserEqual()
> adding to testing pkg, please. Thanks.

As stated, the testing package follows different principles, but you
can use gocheck today for that:

http://labix.org/gocheck

Andrew Gerrand

unread,
Feb 16, 2012, 10:33:37 PM2/16/12
to liigo, golang-nuts
On 17 February 2012 14:04, liigo <com....@gmail.com> wrote:
> +1 for AssertEqual(). It's much easier to use (less code to type) and
> to understand,

Easier to understand than an if statement? Really?

Andrew

Nigel Tao

unread,
Feb 16, 2012, 11:32:28 PM2/16/12
to liigo, golang-nuts
On 17 February 2012 14:04, liigo <com....@gmail.com> wrote:
> +1 for AssertEqual(). It's much easier to use (less code to type) and
> to understand,

http://golang.org/doc/go_faq.html#assertions is relevant, if you
haven't already read it.

Another problem with things like assert.Equal is that it's hard to
agree on what 'equals' means: shallow versus deep, slice/map
references versus contents, tolerance margins for floating point,
dealing with NaN, type differences (is myInt(3) 'equal' to
yourInt(3))? I have already seen a Go assertion package that converted
interface{} to string before comparing, so that int32(3) was 'equal'
to float64(3.0) was 'equal' to the string "3". That's a perfectly
consistent concept of 'equality', but it is a surprising one, whose
semantics are not obvious just from the call site: assert.Equal(foo,
bar). Future maintainers might inadvertently miss false positives.

In comparison, the meaning of '==' is defined by the language
specification, and is something that all Go programmers should be
familiar with. If I need to check a more exotic predicate, I already
have a perfectly good language to describe it: the Go programming
language.

Please try using "if foo != bar" instead of assert.Equal(foo, bar).

Gustavo Niemeyer

unread,
Feb 16, 2012, 11:38:59 PM2/16/12
to Nigel Tao, liigo, golang-nuts
> Another problem with things like assert.Equal is that it's hard to
> agree on what 'equals' means: shallow versus deep, slice/tmap

Equal(s) should mean the same as ==. Anything else is obviously incoherent.

This is something I got wrong early on with gocheck (it does
reflect.DeepEqual), but that will be fixed in the next release.

Liigo Zhuang

unread,
Feb 16, 2012, 11:57:36 PM2/16/12
to Andrew Gerrand, golang-nuts


2012/2/17 Andrew Gerrand <a...@golang.org>
YES. "Equal" is literal meaning, but '"==" is not.

And AssertEqual() do need less code to type, than if statement:


AssertEqual(2, Sum(1,1))  // much easier to use

VS.

if Sum(1,1) == 2 {
    t.Errorf("Not equal ...expect xxx but got xxxx", ...)  // too lengthy
    // if we want print the result of Sum(1,1) here, a new variant is required, or invoke Sum() twicely.
}

Liigo Zhuang

unread,
Feb 17, 2012, 12:45:59 AM2/17/12
to Nigel Tao, golang-nuts


2012/2/17 Nigel Tao <nige...@golang.org>

On 17 February 2012 14:04, liigo <com....@gmail.com> wrote:
> +1 for AssertEqual(). It's much easier to use (less code to type) and
> to understand,

http://golang.org/doc/go_faq.html#assertions is relevant, if you
haven't already read it.

Language assertions is not for testing. And here we are talking about test-framework.
 

Another problem with things like assert.Equal is that it's hard to
agree on what 'equals' means: shallow versus deep, slice/map
references versus contents, tolerance margins for floating point,
dealing with NaN, type differences (is myInt(3) 'equal' to
yourInt(3))?

It's easy to make sure that, AssertEqual() uses the same meaning as ==. So it's NOT hard to agree on.
 
I have already seen a Go assertion package that converted
interface{} to string before comparing, so that int32(3) was 'equal'
to float64(3.0) was 'equal' to the string "3". That's a perfectly
consistent concept of 'equality', but it is a surprising one, whose
semantics are not obvious just from the call site: assert.Equal(foo,
bar). Future maintainers might inadvertently miss false positives.

In comparison, the meaning of '==' is defined by the language
specification, and is something that all Go programmers should be
familiar with.

Again, the AssertEqual() can use just the same meaning as '=='.
 
If I need to check a more exotic predicate, I already
have a perfectly good language to describe it: the Go programming
language.

Please try using "if foo != bar" instead of assert.Equal(foo, bar).

If we talking about language assertions, instead of testing-framework,  it can be writen as: 
assert(foo, bar) // compiler do the works under the hood: check the equal, report assert error (if any), stop compiling if need to do it.
This is another topic.

Nigel Tao

unread,
Feb 17, 2012, 12:52:35 AM2/17/12
to Liigo Zhuang, golang-nuts
On 17 February 2012 16:45, Liigo Zhuang <com....@gmail.com> wrote:
> 2012/2/17 Nigel Tao <nige...@golang.org>

>> http://golang.org/doc/go_faq.html#assertions is relevant, if you
>> haven't already read it.
>
> Language assertions is not for testing. And here we are talking about
> test-framework.

The second paragraph of that FAQ entry starts with: "The same
arguments apply to the use of assert() in test programs." That
paragraph applies equally well to any AssertEquals proposals.

Liigo Zhuang

unread,
Feb 17, 2012, 1:01:43 AM2/17/12
to golang-nuts, Nigel Tao
AssertEqual() ( or t.AssertEqual() ) has another superiority: even if the test-case is not failed, a record can be loged automaticly for later review. So AssertEqual(a,b) equals to:

if a != b {
    t.Errorf(......)
} else {
    t.Logf("we have passed a test-case ...", ...)
}


2012/2/17 Liigo Zhuang <com....@gmail.com>

Liigo Zhuang

unread,
Feb 17, 2012, 1:16:57 AM2/17/12
to Nigel Tao, golang-nuts
2012/2/17 Nigel Tao <nige...@golang.org>

If a real program, not a test program, is runing, and an assert() fails at runtime, what things will occurs? No one knows. So i think all assert()s should be done at compile-time, not runtime. So it's not for testing, IMO.

cheers,
Liigo Zhuang

Nigel Tao

unread,
Feb 17, 2012, 1:26:15 AM2/17/12
to Liigo Zhuang, golang-nuts
On 17 February 2012 17:16, Liigo Zhuang <com....@gmail.com> wrote:
> If a real program, not a test program, is runing, and an assert() fails at
> runtime, what things will occurs? No one knows. So i think all assert()s
> should be done at compile-time, not runtime. So it's not for testing, IMO.

You misunderstand my point. I'm not saying that a built-in assert
function, if it existed, should be used for testing. I am saying that
replacing a language-provided "assert()" with a package-provided
"AssertEquals" in that paragraph gives a relevant argument:

The same arguments apply to the use of AssertEquals in test programs.
Proper error handling means letting other tests run after one has
failed, so that the person debugging the failure gets a complete
picture of what is wrong. It is more useful for a test to report that
isPrime gives the wrong answer for 2, 3, 5, and 7 (or for 2, 4, 8, and
16) than to report that isPrime gives the wrong answer for 2 and
therefore no more tests were run. The programmer who triggers the test
failure may not be familiar with the code that fails. Time invested
writing a good error message now pays off later when the test breaks.

Jan Mercl

unread,
Feb 17, 2012, 2:54:57 AM2/17/12
to golan...@googlegroups.com
On Friday, February 17, 2012 4:04:30 AM UTC+1, liigo wrote:
+1 for AssertEqual(). It's much easier to use (less code to type) and
to understand, it has been widely used in other languages / test-
frameworks, such as JUnit and TestNG. 

Zillion flies can't be wrong, ... you know how it continues ;-)

Jan Mercl

unread,
Feb 17, 2012, 3:03:02 AM2/17/12
to golan...@googlegroups.com
On Friday, February 17, 2012 3:25:23 AM UTC+1, Gustavo Niemeyer wrote:
Hey Andrew,

> AssertEquals provides no information, just a line number and whether
> the assertion was true or false. This call
>
>  AssertEquals(t, foo, bar)
>
> can, at most, say something like "At line x: fooval != barval". Add a

This is somewhat of a pessimistic view on what one can do.

Given this:

    func Quux(s string) string { return "baz" }

    func (S) TestDetails(c *C) {
            c.Assert(Quux("foo"), Equals, "bar")
    }

This is the real output from gocheck

----------------------------------------------------------------------
FAIL: foo_test.go:21: S.TestDetails

foo_test.go:22:
    c.Assert(Quux("foo"), Equals, "bar")
... obtained string = "baz"
... expected string = "bar"


Ignoring the formatting, where is any real semantic difference from Andrew's "can, at most, say something like "At line x: fooval != barval"?

Jamu Kakar

unread,
Feb 17, 2012, 4:25:49 AM2/17/12
to Gustavo Niemeyer, Steven Degutis, golang-nuts
Hi Gustavo,

On Thu, Feb 16, 2012 at 10:43 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
>> ... and so on.  Right now the only issue I have with it is that 'go
>> test' doesn't accept gocheck specific command-line arguments (at
>> least, not in golang-weekly 2012-02-07-1~11764~oneiric1 on Ubuntu),
>> but I guess that's a temporary one.
>
> I don't think the command line arguments ever stopped working. What's
> the issue you're facing there?

Oh, awesome, this turned out to be a case of EPEBKAC. I was passing
-gocheck.v instead of --gocheck.v. I guess I was confused by the 'go
test helpflag' documentation, because the builtin flags all use a
single dash. Anyway, after you said things should work I tried it and
typed two dashes instinctively and it Just Worked(tm)!

You fix bugs without writing code... it's like magic. ;)

Take care,
J.

liigo

unread,
Feb 17, 2012, 4:52:08 AM2/17/12
to golang-nuts
On Feb 17, 2:26 pm, Nigel Tao <nigel...@golang.org> wrote:
> On 17 February 2012 17:16, Liigo Zhuang <com.li...@gmail.com> wrote:
>
> > If a real program, not a test program, is runing, and an assert() fails at
> > runtime, what things will occurs? No one knows. So i think all assert()s
> > should be done at compile-time, not runtime. So it's not for testing, IMO.
>
> You misunderstand my point. I'm not saying that a built-in assert
> function, if it existed, should be used for testing. I am saying that
> replacing a language-provided "assert()" with a package-provided
> "AssertEquals" in that paragraph gives a relevant argument:
>
> The same arguments apply to the use of AssertEquals in test programs.
> Proper error handling means letting other tests run after one has
> failed, so that the person debugging the failure gets a complete
> picture of what is wrong. It is more useful for a test to report that
> isPrime gives the wrong answer for 2, 3, 5, and 7 (or for 2, 4, 8, and
> 16) than to report that isPrime gives the wrong answer for 2 and
> therefore no more tests were run.

easy to do it, if AssertEquals() returns the assert result (bool):

if AssertEquals(...) == false {
break // break for loop to stop more tests, or just return, or
invoke t.Failf(), it's up to you
}

and not forbidding the old way: if a!=b {t.Errorf()} else {}

Gustavo Niemeyer

unread,
Feb 17, 2012, 7:30:42 AM2/17/12
to liigo, golang-nuts
> easy to do it, if AssertEquals() returns the assert result (bool):

With gocheck you can do this instead to do non-terminating verifications:

c.Check(Quux("foo"), Equals, "bar")

> and not forbidding the old way: if a!=b {t.Errorf()} else {}

Right, it allows you to do that as well, when it's more appropriate.

Stop arguing with them and go test some software. :)

Francisco Souza

unread,
Feb 17, 2012, 8:06:03 AM2/17/12
to Andrew Gerrand, liigo, golang-nuts

Yes.

--
~f

spir

unread,
Feb 17, 2012, 1:30:58 PM2/17/12
to golan...@googlegroups.com
On 02/17/2012 05:38 AM, Gustavo Niemeyer wrote:
>> Another problem with things like assert.Equal is that it's hard to
>> agree on what 'equals' means: shallow versus deep, slice/tmap
>
> Equal(s) should mean the same as ==. Anything else is obviously incoherent.
>
> This is something I got wrong early on with gocheck (it does
> reflect.DeepEqual), but that will be fixed in the next release.

+1 for having Equals() mean "==".

A possible extension: have Equals() check equality of structures /
references for Go's referenced types, i.e. slice & map. (This respects
Go's ref semantics for those types. It may be useful that Go defines
"==" as meaning precisely this for slice & map. For slice, this
translates to equality of pointer & len (cap left out). Cheap and useful.)

Denis


Dustin

unread,
Feb 17, 2012, 1:42:53 PM2/17/12
to golan...@googlegroups.com, liigo

On Thursday, February 16, 2012 7:06:56 PM UTC-8, Gustavo Niemeyer wrote:
On Fri, Feb 17, 2012 at 01:04, liigo wrote:
> +1 for AssertEqual(). It's much easier to use (less code to type) and
> to understand, it has been widely used in other languages / test-
> frameworks, such as JUnit and TestNG. I would like to see AsserEqual()
> adding to testing pkg, please. Thanks.

As stated, the testing package follows different principles, but you
can use gocheck today for that:

    http://labix.org/gocheck


  gocheck looks like what I want, but I think it also reveals a weakness in the "go" tool.  If I want to build something that has a transitive dependency that uses this really awesome testing tool, I have to have bzr installed:

launchpad.net/gocheck: exec: "bzr": executable file not found in $PATH

  Users already have to have git and mercurial installed to run this stuff.  These all seem unnecessary.

spir

unread,
Feb 17, 2012, 1:45:05 PM2/17/12
to golan...@googlegroups.com

I completely agree about usefulness of full information reports for
testing --more precisely for diagnosis at development (or modification)
time. Thus, testing checks should not behave like typical asserts which
just stop there. Moreover, they should extensively report on each
failure found, and even possibly on successful checks (eg explicitely
knowing for which combinations of values a bugged func works as expected
is often as useful as knowing for which it fails.)
Things are quite different for regression/maintenance tests, where one
just wants to know whether everything (still) works fine after
compilation on a given system, or after changes in (seemingly) unrelated
parts of an app. In this case, all tests should still run (no break
after a failing check) but information should be minimal. Then a
developper could quickly choose which precise test or testsuite launch,
this time in diagnosis mode.

My dream testing check func(s) would thus have 2 modes, one verbose for
dev/diagnosis, one nearly silent for regression/maintenance. But in both
cases check failures would cause output and go on the tests.

Denis

spir

unread,
Feb 17, 2012, 1:54:49 PM2/17/12
to golan...@googlegroups.com

I would love a compiler-assisted testing check func like:
check(square(-3), 9)

printing on failure:
*** Test Error: check failure in package Foo, function testSquare, l 123
expression : square(3)
expected : 9
result : -9
It needs I guess compiler assistance to give back the original
expression; possibly (haven't yet explored reflection capabilities of
Go) for exact location of call to check().

In diagnosis mode, it may also print out on success succintly:
* check: square(-9) --> 9

Denis

spir

unread,
Feb 17, 2012, 1:58:26 PM2/17/12
to golan...@googlegroups.com
On 02/17/2012 03:25 AM, Gustavo Niemeyer wrote:
> Hey Andrew,
>
>> AssertEquals provides no information, just a line number and whether
>> the assertion was true or false. This call
>>
>> AssertEquals(t, foo, bar)
>>
>> can, at most, say something like "At line x: fooval != barval". Add a
>
> This is somewhat of a pessimistic view on what one can do.
>
> Given this:
>
> func Quux(s string) string { return "baz" }
>
> func (S) TestDetails(c *C) {
> c.Assert(Quux("foo"), Equals, "bar")
> }
>
> This is the real output from gocheck
>
> ----------------------------------------------------------------------
> FAIL: foo_test.go:21: S.TestDetails
>
> foo_test.go:22:
> c.Assert(Quux("foo"), Equals, "bar")
> ... obtained string = "baz"
> ... expected string = "bar"

That's great!
Would you explain (or point to relevant code/doc) how you obtain the
original expressoin? (possibly off list)

Thank you,
Denis

spir

unread,
Feb 17, 2012, 2:03:17 PM2/17/12
to golan...@googlegroups.com
You have the original expression eval'ed, not a plain val result with no
semantic meaning for the developper. We need not
-9 != 9
which says nothing and thus is of no help; we need instead
expression: square(-3) ; expected 9, got -9

Denis

Gustavo Niemeyer

unread,
Feb 17, 2012, 2:18:15 PM2/17/12
to spir, golan...@googlegroups.com
> +1 for having Equals() mean "==".
>
> A possible extension: have Equals() check equality of structures /
> references for Go's referenced types, i.e. slice & map. (This respects Go's
> ref semantics for those types. It may be useful that Go defines "==" as
> meaning precisely this for slice & map. For slice, this translates to
> equality of pointer & len (cap left out). Cheap and useful.)

That's what DeepEquals does, and it will continue to exist with that
name. Equals should choke on that since == will too.

Gustavo Niemeyer

unread,
Feb 17, 2012, 2:20:59 PM2/17/12
to spir, golan...@googlegroups.com
> That's great!
> Would you explain (or point to relevant code/doc) how you obtain the
> original expressoin? (possibly off list)

Using the go/* packages:

http://weekly.golang.org/pkg/go/

Steven Degutis

unread,
Feb 17, 2012, 2:14:46 PM2/17/12
to golang-nuts
I see that you disagree with my particular suggestion of how this
feature could be used. Unfortunately it's inherently part of my
suggestion that I cannot come up with better examples, because the
idea is to allow anyone to experiment and come up with better examples
over time with real, functioning code.

I've been wanting to test that a function results in a specific value.
Currently I test them inline:

if Parse("(foo bar)") != NewList(NewSymbol("foo"), NewSymbol("bar"))
{
t.Errorf("expected (foo bar) to equal %v, but got %v",
NewList(NewSymbol("foo"), NewSymbol("bar")), Parse("(foo bar)"))
}

As you can see, there is a lot of redundancy here. Most of the
redundancy can be reduced by breaking the test into multiple lines,
but that quickly gets cumbersome when there are over a hundred such
tests. And the more I refactor it to be less redundant, the more I see
a function forming, which I simply type over and over because
IndirectlyLogf or something like it doesn't exist:

str, result, expected := "(foo bar)", Parse(str),
NewList(NewSymbol("foo"), NewSymbol("bar"))
if result != expected {
t.Errorf("expected %s to equal %v, but got %v", str, expected,
result)
}

Even if I wanted to write my own private compareParseValue() function,
it couldn't be done without the proposed addition to pkg/testing.

-Steven


On Feb 16, 5:17 pm, Andrew Gerrand <a...@golang.org> wrote:
> Helper functions like AssertEquals are antithetical to the testing
> package's philosophy.
>
> AssertEquals provides no information, just a line number and whether
> the assertion was true or false. This call
>
>   AssertEquals(t, foo, bar)
>
> can, at most, say something like "At line x: fooval != barval". Add a
> new arg to give context:
>
>   AssertEquals(t, "Quux(args)", foo, bar)
>
> so now you have "at line x: Quux(args) == fooval, want barval" which
> is a somewhat informative if rigidly formatted error message. And you
> need to remember the order of arguments for your AssertEquals
> function. This kind of design quickly spirals into dozens of AssertFoo
> variations which aren't much fun to use.
>
> Give me this any day:
>
>   if foo != bar {
>     t.Errorf("Quux(%q) == %q, want %q", args, foo, bar)
>   }
>
> Unless you can propose a more palatable use case for IndirectlyLogf, I
> don't think you'll get much traction with this idea in the core Go
> testing package.
>
> Andrew

Gustavo Niemeyer

unread,
Feb 17, 2012, 2:32:40 PM2/17/12
to Dustin, golan...@googlegroups.com, liigo
> launchpad.net/gocheck: exec: "bzr": executable file not found in $PATH
>
> Users already have to have git and mercurial installed to run this stuff.
> These all seem unnecessary.

I agree. Let's all go back to CVS.

spir

unread,
Feb 17, 2012, 2:34:03 PM2/17/12
to golan...@googlegroups.com
On 02/17/2012 08:18 PM, Gustavo Niemeyer wrote:
>> +1 for having Equals() mean "==".
>>
>> A possible extension: have Equals() check equality of structures /
>> references for Go's referenced types, i.e. slice& map. (This respects Go's

>> ref semantics for those types. It may be useful that Go defines "==" as
>> meaning precisely this for slice& map. For slice, this translates to
>> equality of pointer& len (cap left out). Cheap and useful.)

>
> That's what DeepEquals does, and it will continue to exist with that
> name. Equals should choke on that since == will too.

Hum, I'll explore your package soon. Bit if DeepEquals performs deep
equality as commonly understood, that's not what I meant, rather the
opposite: for slices and maps, just check equality of structures
wrapping the contents, and stops there (no equality xhexk of contents,
even less recusively). It's analog to equality of object refs.
Maybe that's what DeepEquals does as you sat, then great! but if so the
name of the func is /for me/ misleading.

Denis

Paul Borman

unread,
Feb 17, 2012, 2:34:31 PM2/17/12
to Steven Degutis, golang-nuts
I did provide a better use case:

func testErr(t *testing.T, err error, s string) {
        _, _, line, _ := runtime.Caller(1)
        switch {
        case err == nil && s == "":
                return
        case err == nil:
                t.Errorf("%d: Expected error %q, got none", line, s)
        case s == "":
                t.Errorf("%d: Unexpected error %q", line, err)
        case !strings.Contains(err.Error(), s):
                t.Errorf("%d: Got error %q, expected %q", line, err, s)
        }
}

This use case also presents a hacky solution using the existing framework.

    -Paul

Gustavo Niemeyer

unread,
Feb 17, 2012, 2:36:40 PM2/17/12
to spir, golan...@googlegroups.com
> It's analog to equality of object refs.

It's not just analogous. That's exactly what it is: &a == &b works if
both a and b are slices.

Gustavo Niemeyer

unread,
Feb 17, 2012, 2:41:05 PM2/17/12
to Steven Degutis, golang-nuts
> if Parse("(foo bar)") != NewList(NewSymbol("foo"), NewSymbol("bar"))
(...)

> As you can see, there is a lot of redundancy here. Most of the
> redundancy can be reduced by breaking the test into multiple lines,
> but that quickly gets cumbersome when there are over a hundred such
> tests.

This is a good use case for table-styled tests:

var tests := []struct{s string; e Expr}{
{"(foo bar)", NewList(NewSymbol("foo"), NewSymbol("bar")},
}

Put as many entries as you want, write one short for loop.

Kyle Lemons

unread,
Feb 17, 2012, 7:21:13 PM2/17/12
to golang-nuts
At risk of getting mired in this discussion again, I want to point out another problem that I found when I wrote an assertion-based testing framework and started using it.

When you write tests with asserts, you tend to copy and paste them to make new test cases.  This leads to lots and lots of test functions that are slightly different, all of which have to be changed when you change something in the function under test (or find a bug in the code you copied from, or introduced by forgetting to update something you copied).  When you use table driven tests, all of the "test case" information is separate from the "how to test each case" information, and then the latter can be quite verbose and descriptive.   Comparing code for an equivalent Python class to code I wrote in Go had 3x as many lines in the python unit test (which used assertions and mocks) for half the number of test cases.

Steven Degutis

unread,
Feb 17, 2012, 8:24:06 PM2/17/12
to golang-nuts
I should mention that I wrote the gobdd lib, which I prefer over
gocheck or any other third party lib for unit testing in Go. But the
goal I'm reaching for in proposing this change (or at least some
extensibility addition!) is to make use of the built-in testing
package, instead of moving away from it.

The aim here is to reduce duplication as much as possible in my unit
tests, while making it easy to see what failed and where in my console
output. Anything which does this, I'm fine with.

-Steven

Rob 'Commander' Pike

unread,
Feb 17, 2012, 8:48:02 PM2/17/12
to Steven Degutis, golang-nuts

On 18/02/2012, at 12:24 PM, Steven Degutis wrote:

> I should mention that I wrote the gobdd lib, which I prefer over
> gocheck or any other third party lib for unit testing in Go. But the
> goal I'm reaching for in proposing this change (or at least some
> extensibility addition!) is to make use of the built-in testing
> package, instead of moving away from it.
>
> The aim here is to reduce duplication as much as possible in my unit
> tests, while making it easy to see what failed and where in my console
> output. Anything which does this, I'm fine with.

package testing sounds like just the ticket.

-rob


roger peppe

unread,
Feb 17, 2012, 8:56:12 PM2/17/12
to Kyle Lemons, golang-nuts

Yeah, table-driven testing FTW!

Seriously I think launchpad.net/gocheck works very well, and although it does make it easier to write long-winded tests, it also relieves some of the tedium of writing any tests, table-driven or not.

Assert and Check print the values that were compared and the code context, and that's often the only thing you need to diagnose the problem.

I'm not sure about the idiom of importing gocheck to '.' though. I'm not sure I want it *that* special.

    c.Assert(err, C.IsNil)

would read Ok to me if gocheck imported as C.

Nigel Tao

unread,
Feb 17, 2012, 9:12:16 PM2/17/12
to Paul Borman, Steven Degutis, golang-nuts
On 18 February 2012 06:34, Paul Borman <bor...@google.com> wrote:
> func testErr(t *testing.T, err error, s string) {
>         _, _, line, _ := runtime.Caller(1)
>         switch {
>         case err == nil && s == "":
>                 return
>         case err == nil:
>                 t.Errorf("%d: Expected error %q, got none", line, s)
>         case s == "":
>                 t.Errorf("%d: Unexpected error %q", line, err)
>         case !strings.Contains(err.Error(), s):
>                 t.Errorf("%d: Got error %q, expected %q", line, err, s)
>         }
> }

If you tease out the *testing.T arg, and reduce it to
func match(error, string) bool
then you don't need to off-by-one the call stack and double-print line
numbers, and you can also control whether or not a mismatch is fatal:

if err := doSomething(); !match(err, testcase.want) {
t.Errorf("doSomething: got %v want %v", err, testcase.want)
}

Liigo Zhuang

unread,
Feb 17, 2012, 10:39:34 PM2/17/12
to spir, golan...@googlegroups.com

AssertEqual()-like functions with a external configuration (maybe a cmd's arg/flag, an XML or other), can do this well. Just as junit/testng did.

But the currently pkg/testing's if-statement-like tests only report information when test fails. So user don't know explicitly weather or not the tests did really done, if not fails.

Lingo Zhuang

Liigo Zhuang

unread,
Feb 17, 2012, 11:10:16 PM2/17/12
to Kyle Lemons, golang-nuts

In assertion-based tests, you can use table-drive too; in if-statement-based tests, you can copy-paste codes too. NO one tends to write bad codes.

Russ Cox

unread,
Feb 18, 2012, 8:13:33 PM2/18/12
to Steven Degutis, golang-nuts
On Wed, Feb 15, 2012 at 19:28, Steven Degutis <sdeg...@8thlight.com> wrote:
> To give an example use-case where I would like to see it, this is a
> function I have written with it in mind:
>
>        func AssertEquals(t *testing.T, a interface{}, b interface{}) {

The fact that the file:line information in the testing errors
discourages wrappers like this is one of the best things
about them. It is second only to the fact that they produce
useful information that makes debugging broken tests easier.

Russ

minux

unread,
Feb 19, 2012, 2:15:27 AM2/19/12
to Liigo Zhuang, golan...@googlegroups.com
On Sat, Feb 18, 2012 at 11:39 AM, Liigo Zhuang <com....@gmail.com> wrote:

AssertEqual()-like functions with a external configuration (maybe a cmd's arg/flag, an XML or other), can do this well. Just as junit/testng did.

I think the discussion is not about capability (I'm pretty sure these two approaches are equivalent), but about (testing) style.

But the currently pkg/testing's if-statement-like tests only report information when test fails. So user don't know explicitly weather or not the tests did really done, if not fails.

Have you tried 'go test -v'?

Liigo Zhuang

unread,
Feb 19, 2012, 1:24:30 PM2/19/12
to minux, golan...@googlegroups.com

Not the same things. Most tests in std pkgs, are coded as: if a != b { reports errors } No else-statement, so if a == b, nothing information will be shown.

Liigo Zhuang

unread,
Feb 19, 2012, 1:31:09 PM2/19/12
to minux, golan...@googlegroups.com


在 2012-2-19 下午3:15,"minux" <minu...@gmail.com>写道:
>
>

> On Sat, Feb 18, 2012 at 11:39 AM, Liigo Zhuang <com....@gmail.com> wrote:
>>
>> AssertEqual()-like functions with a external configuration (maybe a cmd's arg/flag, an XML or other), can do this well. Just as junit/testng did.
>
> I think the discussion is not about capability (I'm pretty sure these two approaches are equivalent), but about (testing) style.

AssertEqual() do the same things (or a little more) with the same style, it do let users write less codes.

chris dollin

unread,
Feb 19, 2012, 1:55:51 PM2/19/12
to Liigo Zhuang, minux, golan...@googlegroups.com
On 19 February 2012 18:24, Liigo Zhuang <com....@gmail.com> wrote:

> Not the same things. Most tests in std pkgs, are coded as: if a != b {
> reports errors } No else-statement, so if a == b, nothing information will
> be shown.

Passing tests are not particularly interesting, and (hopefully) there
will be a great many of them, so showing no information is the right
choice.

Failing tests are interesting, so showing something is the right choice
for them.

Chris

--
Chris "allusive" Dollin

spir

unread,
Feb 19, 2012, 2:10:05 PM2/19/12
to golan...@googlegroups.com

This is true for regression testing only, where you only want to check
that/whether all (still) works fine. The field of testing is, imo, far
wider than that. During development, i want tests to provide me with all
relevant info telling me how the presently considered/developped piece
of code works, or helping me and diagnose issues I have with it. Passing
checks are as informative as failing ones.

Denis

Jan Mercl

unread,
Feb 19, 2012, 2:23:25 PM2/19/12
to golan...@googlegroups.com
Use t.Log("passed X") and -test.v please.

Liigo Zhuang

unread,
Feb 20, 2012, 8:54:34 AM2/20/12
to Jan Mercl, golan...@googlegroups.com

Just as Danis said, sometimes we need that information, and sometimes not. I don't want to change the test source code every times. AssertEqual() + external configure can do this well.

chris dollin

unread,
Feb 20, 2012, 8:58:07 AM2/20/12
to Liigo Zhuang, Jan Mercl, golan...@googlegroups.com
On 20 February 2012 13:54, Liigo Zhuang <com....@gmail.com> wrote:
> Just as Danis said, sometimes we need that information, and sometimes not. I
> don't want to change the test source code every times.

Hence -test.v

> 在 2012-2-20 上午3:23,"Jan Mercl" <jan....@nic.cz>写道:
>>
>> Use t.Log("passed X") and -test.v please.

Chris

--
Chris "allusive" Dollin

Liigo Zhuang

unread,
Feb 20, 2012, 9:25:14 AM2/20/12
to chris dollin, golan...@googlegroups.com, Jan Mercl


在 2012-2-20 下午9:58,"chris dollin" <ehog....@googlemail.com>写道:
>
> On 20 February 2012 13:54, Liigo Zhuang <com....@gmail.com> wrote:
> > Just as Danis said, sometimes we need that information, and sometimes not. I
> > don't want to change the test source code every times.
>
> Hence -test.v

At least 3 lines code to be add: else-statement.
V.S.
AssertEqual() still that one line with any change.

if a != b {
    t.Error(...)
} else {
    t.Log(...)
}

VS.

AssertEqual(a, b, ...)

Keunwoo Lee

unread,
Dec 2, 2015, 4:33:51 PM12/2/15
to golang-nuts, sdeg...@8thlight.com
I'd like to resurrect discussion of a variation of one small part of this idea.

On Wednesday, February 15, 2012 at 4:28:43 PM UTC-8, Steven Degutis wrote:
So the change I'm proposing is one new and isolated function in
'testing':

        func (c *T) IndirectlyLogf(format string, args ...interface{})

This function would be identical to Logf() except it would print out
the file:line one higher in the stack trace than its existing
counterpart.

For prettifying error messages generated by testing frameworks and the like, I suggest a more modest and compositional addition to testing.go:

func (c *common) Frames(offset int)

which would have the effect of logically "pushing" or "popping" n frames onto the stack searched by testing.decorate. testing.common would have an additional field:

type common struct {
...
offset int
}

which is incremented by the Frames() call.  decorate would then take an additional argument

func decorate(s string, offset int) {
... := runtime.Caller(3 + offset)
...
}

Motivation: When we have multiple implementations of an interface in our codebase, we often want to write a suite of "black box" tests which we apply to every implementation of that interface. The natural way to do this is to extract the sections of the suite into functions, where each function may have a fairly detailed set of assertions it makes against a testing.T:

interface Foo { ... }

func FooTestSuite(f Foo, t *testing.T) { ... }

The main foo_impl_test.go file would the instantiate a particular implementation, then run the black box tests, plus additional white box tests:

func TestBarFoo(t *testing.T) {
b := BarFoo{...}
FooTestSuite(b, t)
... // extra white box tests specific to BarFoo
}

The downside of this approach is that all assertions are then annotated with the line number of the call to FooTestSuite(), not the site of the test failure itself.

Now, we could somewhat laboriously plumb error messages from the test suite to the main test:

func FooTestSuite(f Foo) []error

func TestBarFoo(t *testing.T) {
b := BarFoo{...}
errs := FooTestSuite(b)
for _, e := range errs {
t.Error(e.Error())
}
}

but this is more cumbersome than just invoking the testing.T functions that we would invoke if the tests were inlined at the call site.  testing.BT is a convenient, expressive interface, and it would be nice to use it directly.

With the proposed Frames() method, we can write the test suite as follows:

func FooTestSuite(t *testing.T) {
t.Frames(1)
defer t.Frames(-1)
...
}

Alternatively, you could have PushFrame() and PopFrame(), or we could add a method to TB

type TB interface {
...
PushFrame() TB
}

which returns a copy of the receiver with offset incremented by 1.  The idiomatic usage would then be

func TestBarFoo(t *testing.T) {
b := BarFoo{...}
FooInterfaceTestSuite(b, t.PushFrame())
}

IMO this is actually the cleanest-looking solution from the client's point of view, as the "unwinding" is automatic, although it imposes the burden on TB implementations of cloning themselves.

~k

Reply all
Reply to author
Forward
0 new messages