Non-official alternate API + FAST unit testing apparatus for App Engine Go

[Robbie Iannucci]

Hello!

I'm a member of the Chrome Infrastructure team at Google, and we've been using Go on App Engine more and more for our continuous integration services that we're building for the Chrome/Chromium team.

Along the way, we've developed a sort of alternate API for the datastore, memcache and taskqueue services, focused on extensibility and testing. In particular, we wanted our tests to be really really fast, but also model App Engine's consistency guarantees as accurately as possible. We also missed some of our old Python App Engine tricks like transparent memcaching of certain datastore models.

So, we've developed https://github.com/luci/gae for our own use, and thought it might be useful for others as well. It has a long laundry list of features, but I'll try to stick to the highlights:

• "fake" multi-goroutine-safe datastore, memcache and taskqueue implementations which are pure go (no dev_appserver.py!)
• include test-controllable APIs for deterministically controlling the state of the 'eventually consistent' portions of the datastore for flake-free eventual-consistency tests.
• Implements all public APIs with high accuracy, including transactions and queries (with test-control over compound indexes, and automatic zig-zag query merges when appropriate). 
• Overhead for new test instances of these implementations is ~5 microseconds on commodity hardware (mac laptop). We currently have about 600 test cases which take a total of 3 seconds to run (YMMV).
• Ability to add user-supplied 'filters' which interpose between the exposed service API, and the backend implementation. luci/gae currently includes the following filters:
• dscache - an ndb-style memcache filter; datastore operations will transparently use memcache to accelerate datastore access (inspired by https://github.com/qedus/nds)
• txnBuf - a transparent transaction buffer which allows arbitrary recursive transactions, and will cause Get and Query operations to reflect all previous Delete and Put operations.
• count - a very simple filter which simply counts the number of times a service API was hit
• featureBreaker - selectively cause certain service APIs to return errors, useful for exercising failure cases in tests.
• none of these filters have any special support; you can implement your own filters just as easily to do... whatever you like :).
• "Quality of life" API modifications:
• PropertyConverter interface allows a user type to implement it's own to/from Property serialization; serialize json as compressed []byte, round trip a complex number, or a uint32, etc.
  
• Kind/ID/ParentKey embeddable inside of datastore model (inspired by https://github.com/mjibson/goon), to avoid the need to carry around parallel lists of keys and values.
• context.Context only needs to be passed once per function body to obtain e.g. a datastore.Interface, instead of once per API call.
• PropertyList is now a PropertyMap, which removes the need for the `Multiple` Property data, and makes it a bit easier to implement PropertyLoadSaver.
• all service implementations are completely replaceable at test time.
• Convenient `MakeKey("ParentKind", "stringID", "ChildKind", 100)` method for manufacturing *Keys :)
• 'Raw()' versions of the APIs to bypass all the fancy reflect code if you really want to just manipulate *Keys and PropertyMaps for moar speedz (tm).

And more to come :). Please note that this is not really supported in any way by anyone, but we'd love to get feedback (issues!) on it and take feature improvements (patches!). In particular, we reserve the right to change the interfaces on master at any time for any reason (up to and including temporary or permanent insanity :), so if you do use this, then please, please, please pin it using Goop or godeps or something like that. That said, the interfaces haven't changed in at least a month, so therefore it is very stable in internet-time ;).

I hope someone finds this useful!

Cheers,

Robbie

[Ronoaldo José de Lana Pereira]

Sounds very interesting :-) thanks for sharing!!!

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Probst]

Nice. Way back I looked at the Go APIs and thought "It'd be great if somebody wrote an API that encapsulates all of this in interfaces to make it testable". And you did much more!

I'm poking a bit at this, migrating my trivial blog app (my common guinea pig). A couple of questions:

• Your datastore seems to replace the explicit return of keys with always using the key fields in structs. That's indeed much saner, woohoo. But are there any docs on how the structs should look exactly? Will it just take whatever property of type datastore.Key it finds, or does it need tagging in a specific way?
• What's the best way to handle APIs that have no equivalent (yet), like the mail, user, or log packages? Naively calling with a context I either get nil pointer derefs or "not an App Engine context".
• Are there docs on migrating from the plain vanilla App Engine APIs?

Thanks,

Martin

[Robbie Iannucci]

Hey Martin :)

Some answers:

Q1. "are there any docs on how the structs should look exactly?"

The struct format is explained in the GetPLS docstring (which I realize totally un-findable :)). https://godoc.org/github.com/luci/gae/service/datastore#GetPLS . I filed a bug to make this documentation more prominent (https://github.com/luci/gae/issues/18). I would also point out that the datastore.Interface also contains a 'Raw' version that deals solely with []*datastore.Key and []datastore.PropertyMap, which is occasionally more convenient than the normal fancy interface. TL;DR for this question is that struct fields can be tagged like:

type MyStruct struct {
  // kind is automatically derived from the struct name, but can be overridden

  Kind string `gae:"$kind"`  // allows a dynamically settable kind; OR

  Kind string `gae:"$kind,defaultName`  // default assigned kind name if Kind == ""

  ID int64 `gae:"$id"`  // int id; OR

  ID string `gae:"$id"` // string id

  // ID can have defaults too

  Parent *datastore.Key `gae:"$parent"` // what you think :)

}

This also can stack with the https://godoc.org/github.com/luci/gae/service/datastore#PropertyConverter interface:

type MyFancyID struct {

  Name string

  Category string

}

var _ datastore.PropertyConverter = (*MyFancyID)(nil)

func (i *MyFancyID) ToProperty() (datastore.Property, error) {
  return datastore.MkProperty(fmt.Sprintf("%s|%s", i.Name, i.Category)), nil

}

func (i *MyFancyID) FromProperty(p datastore.Property) error {

  // something like:

  // i.Name, i.Category = strings.SplitN(p.Value().(string), "|", 2)

  // but in the super-verbose go way that handles errors and such :)

}

type MyStruct struct {

  MyFancyID `gae:"$id"`  // Name and Category will be used/filled when writing/reading this from the datastore

}

There's also the full-crazy interface that *MyStruct may implement, too (https://godoc.org/github.com/luci/gae/service/datastore#MetaGetterSetter), which allows MyStruct to completely control all metadata fields via methods.

In general, the metadata (the $blah-tagged fields) concept is extended to filters too. For example, the dscache filter makes use of metadata fields to control cache behavior (https://godoc.org/github.com/luci/gae/filter/dscache#hdr-Cache_control).

Q2. "best way to handle APIs that have no equivalent"

This is a bit mixed. In the 'prod' implementation, the context is compatible with the regular SDK context. In the 'memory' implementation, there's just no implementation of these services (yet; filed https://github.com/luci/gae/issues/19 and https://github.com/luci/gae/issues/20).

The 'log' service is actually implemented elsewhere (https://godoc.org/github.com/luci/luci-go/appengine/gaelogger and https://godoc.org/github.com/luci/luci-go/common/logging/memlogger), because we implement our (yet-another :C) logging wrapper here (https://godoc.org/github.com/luci/luci-go/common/logging/) because we needed a single interface that could work for appengine and also not-appengine code.

Q3. Migration docs?

Unfortunately no :(. We haven't had the opportunity to sit down and think about this process. I added a bug to doc this too :)

Thanks for taking a look!

R

[Ian Rose]

This looks really interesting and potentially very useful.  Unit testing with goapp has been a bit of a struggle for us.  Thanks for sharing!

Are there any issues with mixing usage of this package with the "classic" app engine packages?  What I'm envisioning is that we certainly wouldn't want to convert our entire codebase wholesale.  Instead we might try out this new package in the next one or two files that we write.  But that, of course, means that the datastore entities (or memcache entries, or tasks, or whatever) would sometimes be accessed from these new files (using the new package), and other times from our old files (using the old packages).  Is that kosher?

Thanks again,

- Ian