Testing App Engine stubs

[Rodrigo Moraes]

Hey,

How have you been testing code that uses App Engine APIs (datastore,
memcache, users etc)? Does anybody have some helpers and can share?

(I saw a thread from May about this but it didn't have a conclusion)

-- rodrigo

[Rodrigo Moraes]

On a related note: any chances that the package "appengine_internal"
is made available? I was intending to create a testbed package much
like what we have in python, but it seems that I'd have to reverse-
engineer appengine_internal for that.

-- rodrigo

[Rodrigo Moraes]

Never mind, I found where it is:

http://code.google.com/p/appengine-go/source/browse/appengine_internal/api_dev.go

I didn't know this project existed. :D

-- rodrigo

[Paddy Foran]

Thanks for sharing. Amazingly useful.

Thanks,
Paddy Foran

[David Symonds]

Keep in mind that the appengine_internal packages are all truly
internal. We can change them without notice and break your apps if you
depend on them. Only the appengine packages ("appengine",
"appengine/datastore", etc.) are supported.

Dave.

[Rodrigo Moraes]

Ah, sure. I just needed it because "appengine" packages imports
"appengine_internal", and so it is a dependency for testing.

-- rodrigo

[Brad Fitzpatrick]

I'm dealing with this issue now with my project.  My plan is to run all my App Engine tests in a child process running dev_appserver.py --skip_sdk_update_check --high_replication --clear_datastore etc etc in a temp directory, and then having the parent process hit various URLs in a fake testing app.  Or something.

I think that's the only way to do it, since the datastore, memcache, etc local implementations are all implemented in Python, anyway.

[Johan Euphrosine]

When we discussed this with Rodrigo on #appengine,

We discussed the idea of mocking up context call like this:
c.Call("datastore_v3", "Get", req, res, nil)

by implementing a different Context interface:
http://code.google.com/p/appengine-go/source/browse/appengine/appengine.go

Do you think it would be practicable ?

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[David Symonds]

On Fri, Oct 21, 2011 at 7:48 PM, Johan Euphrosine <pro...@google.com> wrote:

> When we discussed this with Rodrigo on #appengine,
>
> We discussed the idea of mocking up context call like this:
> c.Call("datastore_v3", "Get", req, res, nil)
>
> by implementing a different Context interface:
> http://code.google.com/p/appengine-go/source/browse/appengine/appengine.go
>
> Do you think it would be practicable ?

It's practical, but not a particularly useful way to test your code.
The test would be covering implementation details of the
appengine/datastore package. A better approach would be to make a tiny
interface for the aspects of appengine/datastore that you use, have
one real implementation of it that just plumbs through to
appengine/datastore, and then use that interface in your code. Then
you can mock out that interface in your tests.

Dave.

[Johan Euphrosine]

Maybe an approach similar to appengine.ext.testbed for python is also possible:
http://code.google.com/appengine/docs/python/tools/localunittesting.html
http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/testbed/__init__.py

But that would mean require implementing datastore stub similar to
what is in the python SDK:
http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/api/datastore_file_stub.py

[Rodrigo Moraes]

On Oct 21, 6:52 am, David Symonds wrote:
> It's practical, but not a particularly useful way to test your code.
> The test would be covering implementation details of the
> appengine/datastore package. A better approach would be to make a tiny
> interface for the aspects of appengine/datastore that you use, have
> one real implementation of it that just plumbs through to
> appengine/datastore, and then use that interface in your code. Then
> you can mock out that interface in your tests.

I'm not sure I understand. That would mean creating mocks for in-
memory datastore for queries, gets, puts, deletes, transactions... if
you use all of them?

-- rodrigo

[David Symonds]

You don't have to write a functional mock (called a "fake" in some
circles); you could just have a stub implementation, or use something
like gomock (http://code.google.com/p/gomock/).

If you care enough about the behaviour to warrant a functional mock,
you'd be better off starting up the dev_appserver, because there's a
surprising amount of complexity and edge cases to consider for
something like datastore.

Dave.

[Rodrigo Moraes]

On Oct 21, 11:53 pm, David Symonds wrote:
> If you care enough about the behaviour to warrant a functional mock,
> you'd be better off starting up the dev_appserver, because there's a
> surprising amount of complexity and edge cases to consider for
> something like datastore.

Yes, that's what I thought. That would be insane.

Using dev_appserver is what I want to do. Brad mentions "run App
Engine tests in a child process...", but I'm a little lost how this
would be done. Can you give me a more detailed hint about how this
would work?

-- rodrigo

[Ugorji Nwoke]

I looked into a plan for testing a while ago, and decided to use the rpc package. It's looks like it would work.

http://golang.org/pkg/rpc/

- Write all my tests in a set of test packages, according to the specifications of the rpc package

- Write a TestRunner which would use reflection to see the exported functions of my test packages (ie my test functions) and run them (re-starting or ensuring dev server is running)

- Each time I need a clean slate, I can restart the dev server.

Conceptually, it seemed like it would work. 

I also thought about wrapping the RPC calls into testing.InternalTest[] and using the testing framework as a test runner (if I thought it would provide some extra value). 

Note that I haven't really done any implementation yet - just the general idea when I looked into how I would test the parts of my app that require app engine services.

[Kyle Lemons]

Using dev_appserver is what I want to do. Brad mentions "run App
Engine tests in a child process...", but I'm a little lost how this
would be done. Can you give me a more detailed hint about how this
would work?

One way would be to have a "test" application that emulated the gotest-generated _testmain (so you can reuse the "testing" infrastructure) that first started up the dev_appserver and then (after making sure it was receiving requests) ran the test functions; they would construct HTTP requests to the dev_appserver address (for best results, you'd probably want to choose a random ephemeral port or something) and validated them against their output.  This would be blackbox testing.  There may be a way to use the _ah directory to inspect some internal state (datastore, etc) in the dev_appserver, but I don't know that for sure.  One useful thing about this is that you can emulate task queue, channel, etc events with requests to the appropriate URLs.

~K

[Rodrigo Moraes]

On Oct 22, 3:55 pm, Ugorji Nwoke wrote:
> I looked into a plan for testing a while ago, and decided to use the rpc
> package. It's looks like it would work.
>
> http://golang.org/pkg/rpc/

This sounds like a good plan for me, at least for the time being.

I have a very noob question related to this: how to you access the
current http.Request in a rpc method (so that you can get the App
Engine context)?

-- rodrigo

[Ugorji Nwoke]

Someone else can probably give a more detailed response, but in summary (from looking through the GO Code):

  hdr = http.Header{}

  header.Add("X-AppEngine-Inbound-AppId", "anything")

  req := &http.Request { Header: hdr }

And pass that in.

It's also possible that on the dev server, the actual appid is never really used, so you might be able to get away with just:

  req := &http.Request { Header: http.Header{} }

BTW, I've prototyped the solution using rpc. I changed things a bit but it seems like it works fine.

- Now tests can be server side (ie on the app engine server) or client side (when no app engine services are required)

- When run server side, I just ship the testing.T and the "object" to the rpc handle, and the test is run on there and testing.T is updated, which I see on the runner. 

- With this, tests do not have to know about rpc package. I can just use a runner to run them on the rpc server. 

Mocking things like the request are still necessary, but with GO, mocking them seems really easy.

[Rodrigo Moraes]

On Oct 24, 10:35 am, Ugorji Nwoke wrote:
> Someone else can probably give a more detailed response, but in summary
> (from looking through the GO Code):
>   hdr = http.Header{}
>   header.Add("X-AppEngine-Inbound-AppId", "anything")
>   req := &http.Request { Header: hdr }
>
> And pass that in.

Thanks. Can someone confirm that this is this would be enough to for
production? Basically, I want to know how to get the App Engine
context from a rpc method (need the current http.Request from inside
the method).

Not sure if in production I'll be allowed to fake the "X-AppEngine-
Inbound-AppId" header.

-- rodrigo

[Brad Fitzpatrick]

On Mon, Oct 24, 2011 at 5:43 AM, Rodrigo Moraes <rodrigo...@gmail.com> wrote:

On Oct 24, 10:35 am, Ugorji Nwoke wrote:
> Someone else can probably give a more detailed response, but in summary
> (from looking through the GO Code):
>   hdr = http.Header{}
>   header.Add("X-AppEngine-Inbound-AppId", "anything")
>   req := &http.Request { Header: hdr }
>
> And pass that in.

Thanks. Can someone confirm that this is this would be enough to for
production?

No, it will not.  In production there are other headers you need which can't be predictably faked like that.  (they're large random numbers assigned by App Engine for each incoming request).

 

Basically, I want to know how to get the App Engine
context from a rpc method (need the current http.Request from inside
the method).

Hm, I think the rpc package would need a change to accomodate this.  I had to do something similar with some code I ported to App Engine.

[Ugorji Nwoke]

Just wanted to add some more info on using rpc.

There are some challenges to using rpc built atop the testing framework

- For tests, testing.T fields are not exported, so cannot be shipped across the wire and maintain state (gets zero'ed out). You can work around it by using a diff object across the wire.

- For benchmarks, testing.B is all together not rpc friendly. The result of a test run is not included in B, fields are not exported, etc.

- All messages are logged at the target process, so you cannot easily get the results printed at the test runner (client) without jumping through some hoops

In summary, it seems the testing package was designed for in-process testing, at least that is the case now.

Unless something changes in the testing package, the model I've used now which has worked is to skip rpc altogether and run the tests locally in the app engine process. This seems to work from my initial tests.

- use flag.Set(...) to set the flags you want the testing package to honor

- call testing.Main(...) // I have a light layer over this that can use reflection to construct InternalTest[] and InternalBenchmark[] for methods in types which match TestXXX or BenchXXX respectively. and call testing.Main(...) with these. 

If anyone figures out a way to make rpc work with the testing package without jumping through hoops for tests and benchmarks, please let me (us) know.

[Brad Fitzpatrick]

I've now done this.  More polish work remains, but it works now:

      https://code.google.com/p/gae-go-testing/

Enjoy.

[Brad Fitzpatrick]

[+list, accidentally dropped]

Good questions.

Currently to develop this I had to install the r60 release, goprotobuf, the appengine-go project, and write a Makefile. Definitely not ideal.

As Go 1 approaches and Makefiles die, we should try to figure out how to make this easy.

Maybe we can move this code into the App Engine SDK somehow.  The uploader could just skip uploading *_test.go files, as is the convention.

On Thu, Nov 3, 2011 at 8:54 AM, Jeff Huter <jbh...@gmail.com> wrote:

Brad,

The Go GAE testing process is unclear to me.  My Go application includes several packages mine as well as 3rd party.  All of which reside in a common directory, the GAE SDK compiles and runs these on demand without any Makefiles.  Moreover, the GAE SDK does not install these packages to the SDK directory.  In order to test, do I need to install Go separately and install each of the packages to the separate Go directory.  Also, do I need to write makes files in order to test?  Basically, I'm a bit confused regarding the best practices for testing GAE applications and how it relates to the SDK.  Moreover, assuming additional files are needed for testing, what is the best way to keep these from being uploaded to production.

Jeff