Suggestions for mocking/stubbing/faking cgo binding?

[Nathan Youngman]

Hi,

I'm learning how to write C bindings in Go, and having some trouble figuring out how to unit test it. So far I've determined to have a low-level package that does a minimal binding. My hope is that I could stub/mock out those calls when unit testing a higher level call. Here's an example of calling glGetError() in OpenGL:

package gl

// #include <OpenGL/gl3.h>
import "C"

type ErrorCode C.GLenum

func Error() ErrorCode {
  return ErrorCode(C.glGetError())
}

Now I'm working on higher level error reporting in my gfx package, which calls gl.Error().

For testing I would want to stub/fake out Error() so that I don't need to call into the OpenGL stack. I was looking at http://golang.org/src/pkg/net/smtp/smtp_test.go?h=fake but I'm still at a loss. I'm still fairly new to Go and interfaces and only just started using gocheck.

Ideally I would use something like gomock/mockgen to ensure Error() is called x number of times, having it return certain values, and then Asserting that the higher level function did the right thing. I'm looking at the somewhat sparse documentation and the sample user mock, but I still don't have the foggiest idea how to structure my code/tests.

Any tutorials or code samples that would help?

Thanks!
Nathan.

[Kyle Lemons]

Even though I wrote one for giggles, I actually haven't found much use for mock generators myself.  They seem to me like they're often used with assertion-style frameworks, and this mailing list long ago taught me why not to use those :).  In any case, you would need to create an interface that mimicked a type (you cannot "stub" out a package in Go) and that interface could either be backed by the underlying package or by your testing mock.  It looks like gl.Error() is a package-level function; one approach would be to have a Context-style interface and object on which you call such methods, so that they aren't rooted in the package.

[Nathan Youngman]

Hi Kyle,

I was resistant to "complicating things" by adding a Rendering Context, but the more I think about it, this sounds like the way to go. In fact, an RC could be useful for the production code, as it could be a struct that caches state.

So let's see if I got this. I need to setup a RenderingContext and a FakeRenderingContext, which higher level functionality would make use of (rc, fakerc). Then I need an interface to satisfy the type checker when passing an rc in. Code that was calling the gl package directly would instead need to use the rc.

  type Errorable interface {

    Error()

  }

  type Error map[gl.ErrorCode]bool // too many things called Error! :-)

  // Get Error

  func GetError(rc Errorable) Error {

    err := Error{}

    // "glGetError should always be called in a loop, until it returns GL_NO_ERROR"

    for code := rc.Error(); code != gl.NoError; code = rc.Error() {

      err[code] = true

    }

    if len(err) > 0 {

      return err

    }

    return nil

  }

  type FakeRenderingContext struct {

  }

  func (rc FakeRenderingContext) Error() ErrorCode {

    return gl.NoError  // or something

  }

In reality, I imagine GetError() above would receive the RC indirectly through some context of its own. Then I can use the FakeRenderingContext to stub out the low-level functionality. This struct could contain some state, if I wanted to return a different error each time, or count how many times GetError() called Error(). In this case the behaviour should be that Error() is called "number of errors + 1" times.

I'm not sure exactly what you mean by "assertion-style frameworks" and why they are best to avoid? 

I've heard of two styles of testing. One with mocks that would verify the "messages", eg. Error() is called twice. The other way I've heard of is to assert state, such as that the return value is nil.

But I wouldn't consider myself an accomplished Tester. I think the main trouble I'll have is how to scale this up, whether to have lots of different FakeRCs and small interfaces, or to fake out the entire GL stack and try to customize that with delegation. That, and just how to write good tests that test the right things!

Thanks for pointing me in the right direction, and for getting me thinking.

Nathan.

On 21 August 2012 16:02, Kyle Lemons <kev...@google.com> wrote:

Even though I wrote one for giggles, I actually haven't found much use for mock generators myself.  They seem to me like they're often used with assertion-style frameworks, and this mailing list long ago taught me why not to use those :).  In any case, you would need to create an interface that mimicked a type (you cannot "stub" out a package in Go) and that interface could either be backed by the underlying package or by your testing mock.  It looks like gl.Error() is a package-level function; one approach would be to have a Context-style interface and object on which you call such methods, so that they aren't rooted in the package.

--
Nathan Youngman
Email: n...@nathany.com
Web: http://www.nathany.com

[Kyle Lemons]

Basically what I'm suggesting is to go from

func SetShader(whatever)

to

func (ctx *GlContext) SetShader(whatever)

and then you have an interface 

type Context interface {

  ...

  SetShader(whatever)

  ...

}

and your testable code uses a Context object, which normally is backed with an instance of GlContext.  In tests, you'd replace it with a test-specfiic mock object that adheres to the interface to do whatever you need for the test in question.

Since there are so dang many functions in OpenGL though, I think this solution may not be scalable.  I missed or forgot that you were wrapping this directly around the cgo stuff.  Functional testing of that sort of interface is probably a good idea, but unit testing is probably not as necessary.  When you are bundling it together into a wrapper library with its own logic that's not handed straight through to cgo, you may need to employ some of these techniques.

On Wed, Aug 22, 2012 at 11:18 AM, Nathan Youngman <n...@nathany.com> wrote:

Hi Kyle,

I was resistant to "complicating things" by adding a Rendering Context, but the more I think about it, this sounds like the way to go. In fact, an RC could be useful for the production code, as it could be a struct that caches state.

So let's see if I got this. I need to setup a RenderingContext and a FakeRenderingContext, which higher level functionality would make use of (rc, fakerc). Then I need an interface to satisfy the type checker when passing an rc in. Code that was calling the gl package directly would instead need to use the rc.

  type Errorable interface {

    Error()

  }

  type Error map[gl.ErrorCode]bool // too many things called Error! :-)

  // Get Error

  func GetError(rc Errorable) Error {

    err := Error{}

    // "glGetError should always be called in a loop, until it returns GL_NO_ERROR"

    for code := rc.Error(); code != gl.NoError; code = rc.Error() {

      err[code] = true

    }

    if len(err) > 0 {

      return err

    }

    return nil

  }

  type FakeRenderingContext struct {

  }

  func (rc FakeRenderingContext) Error() ErrorCode {

    return gl.NoError  // or something

  }

In reality, I imagine GetError() above would receive the RC indirectly through some context of its own. Then I can use the FakeRenderingContext to stub out the low-level functionality. This struct could contain some state, if I wanted to return a different error each time, or count how many times GetError() called Error(). In this case the behaviour should be that Error() is called "number of errors + 1" times.

I'm not sure exactly what you mean by "assertion-style frameworks" and why they are best to avoid? 

Assertion-style frameworks, like you have frequently in C++, Java, and Python, tend to end up with many very similar test cases that have been created via copy-and-paste.  Whenever something fails, the testcase ends and it moves on to the next one.  This makes tests error prone (copy/paste is hard, how many times have you forgotten to change something after pasting?) and can limit the amount of useful information you get (assert errors tend to be pretty generic and you don't see what other failures might happen that could point more directly at a problem if an earlier assertion fails).  In Go, one typically sees table-driven tests with hand-crafted comparisons.  To me, both the code and the errors are a major improvement.

[Nathan Youngman]

Hi Kyle,

I think we're on the same page. I agree, scalability could definitely be an issue, though we'll see if I get so far as to cover the entire OpenGL API. At least I'm only doing Core profile. ^_^

Speaking of shaders, so far I've followed banthar's approach of having a gl.Shader object, so it effectively already has a "context". I should be able to build out a "FakeShader" and then use one or the other to test a higher level (yet-to-be-built) gfx.ShaderManager, that I'm currently modelling after GLTools from the SuperBible.

https://github.com/nathany/mantle/blob/master/core/gl/shader.go

API design is hard.

It's hard to decide whether to make the low-level binding more idiomatic in terms of error handling and such, or to stay true to OpenGL. I think testability could inform that though. For testing, any relatively complex logic should be at a higher level so that I can actually write unit tests.

Since I'm learning OpenGL as I go, I haven't gotten to the point of creating an offscreen framebuffer. I imagine that would be useful for functional testing. Lots to learn!

Take care,

Nathan.

On 22 August 2012 12:34, Kyle Lemons <kev...@google.com> wrote:

Basically what I'm suggesting is to go from

func SetShader(whatever)

to

func (ctx *GlContext) SetShader(whatever)

and then you have an interface 

type Context interface {

  ...

  SetShader(whatever)

  ...

}

and your testable code uses a Context object, which normally is backed with an instance of GlContext.  In tests, you'd replace it with a test-specfiic mock object that adheres to the interface to do whatever you need for the test in question.

Since there are so dang many functions in OpenGL though, I think this solution may not be scalable.  I missed or forgot that you were wrapping this directly around the cgo stuff.  Functional testing of that sort of interface is probably a good idea, but unit testing is probably not as necessary.  When you are bundling it together into a wrapper library with its own logic that's not handed straight through to cgo, you may need to employ some of these techniques.

[Nathan Youngman]

Oh, I missed this in my last reply.

Table driven tests look interesting. I recall seeing something similar in Clojure, and Cucumber has tables as well. I guess it only makes sense, having well factored tests with good error messages. Gocheck does have Commentf() to make the error messages more specific, but perhaps it's just too easy to use the defaults. Thanks for the link.

Nathan.

[roger peppe]

On 22 August 2012 19:34, Kyle Lemons <kev...@google.com> wrote:
> Assertion-style frameworks, like you have frequently in C++, Java, and
> Python, tend to end up with many very similar test cases that have been
> created via copy-and-paste. Whenever something fails, the testcase ends and
> it moves on to the next one. This makes tests error prone (copy/paste is
> hard, how many times have you forgotten to change something after pasting?)
> and can limit the amount of useful information you get (assert errors tend
> to be pretty generic and you don't see what other failures might happen that
> could point more directly at a problem if an earlier assertion fails). In
> Go, one typically sees table-driven tests with hand-crafted comparisons. To
> me, both the code and the errors are a major improvement.

One can combine both techniques. We use launchpad.net/gocheck (an
assertion-style
framework) *and* table-driven tests where appropriate. Works well for us.

For mocking, we never replace public functions. Sometimes, we'll
change a constant to a variable,
so that a test can modify it. It's nice to avoid complicating the code for tests
if possible.