deferred binding for a/b and cohort testing framework - what do you think

[Nick Siderakis]

I'm thinking about building a small framework similar to cohorts.js for GWT that takes advantage of deferred binding.

Does this seem like a good idea?

( cohorts.js https://github.com/jamesyu/cohorts )

[Nick Siderakis]

Some reasons why this is a bad idea

• Does not scale well for many tests (permutation explosion)
• Would not support dynamic (in app) cohort assignment

runAsync seems like a better choice.

[Colin Alworth]

With soft-permutations though, this could cut back on the explosion, though you are right that the app would be unable to switch at runtime between which version it is running. That said, once a user has been set to a particular set of features, you probably want to keep them there. And a runAsync split point for each possible feature will have its own kind of explosion issues, affecting the user more than the compiler though.

Deferred binding lets you replace whole classes, which is probably why you started down this road in the first place. The lib you linked to appears to hook into Google Analytics for collecting data, and leaves the cohort selection to the browser's own random pick. Building this in GWT, I'd probably want to take the approach of making some of these decisions the server's problem - writing out one or more meta tags to help guide permutation (soft or hard) selection lets you tie it to session or user (consistent across page load, across browsers/devices). Additionally, when you make a decision on which implementation to use, simply remove whatever is auto-selecting an implementation in your module file.

Interfaces or base classes would be defined in your code, along with 1 or more implementations. A implementation would be requested in code via GWT.create(MyBaseClass.class), and rules established to decide which to pick. The basic idea behind those rules would be to define the property and its values, and what each value would cause to have happen.

<!-- declare the name of the feature, possibly how it is selected -->
<define-property name="ab.featureOne" values="a,b,c" />

<!-- defaults to selecting via meta tag with name="gwt:property"
     and content="ab.featureOne=X" where X is a, b, or c. This can
     be modified via a property-provider tag to either describe the js
     used to select a property, or to point at a java file that will build
     that js. -->

<!-- Suppress additional permutations - this is optional, and results
     in the final js build being larger than it might otherwise be, but
     with fewer permutations. runAsync can still be used to push all
     of these into another file - but use runAsync normally, don't
     deliberatly push these out unless needed (very large or rarely
     used). -->
<collapse-property name="ab.featureOne" values="*" />

<!-- next, declare rebind instructions -->
<replace-with class="my.package.client.WidgetA">
  <when-type-is class="my.package.client.FeatureOne" />
  <when-property-is name="ab.featureOne" value="a" />
</replace-with>
<replace-with class="my.package.client.WidgetB">
  <when-type-is class="my.package.client.FeatureOne" />
  <when-property-is name="ab.featureOne" value="b" />
</replace-with>
<replace-with class="my.package.client.WidgetA">
  <when-type-is class="my.package.client.FeatureOne" />
  <when-property-is name="ab.featureOne" value="a" />
</replace-with>

There are lots of options to further customize this, as that is a lot of XML just to specify three classes to try out (and we still have to set the page up to write out that meta tag). That being said, there are several ways we can make this simpler if you plan to use it a lot:
Get rid of the replace-with rules:
Define the test to run (ab.featureOne) and the possible options (a,b,c), but build a generator to deal with doing the rebind replacements. Easiest way for this would probably be to add a few annotations and a marker interface:
public interface ABTestable { }
public @interface ABTestName { String value(); }
public @interface ABTestOption { String value(); }

The generator would be defined to build any class that implements ABTestable, and would look around for any possible impls, and pick one based on the option annotation it is decorated with. This xml would then replace the <replace-with> tags for all tests and all options - only needs to be declared once:
<generate-with class="my.package.rebind.ABOptionGenerator">
  <when-type-is class="my.package.client.ABTestable" />
</generate-with>

And your code would look something like this:
@ABTestName("ab.featureOne")
public interface FeatureOne extends ABTestable { ... }

@ABTestOption("a")
public class WidgetA implements FeatureOne { ... }
etc.

And once testing is complete, a single replace-with (or gin bind().to()) can be used, and the AB* annotations and marker interfaces removed.

Better selection:
Instead of always writing out meta tags with the name gwt:property, the bootstrap js file can read from any data available on page load - your own meta tags, the url string itself, cookies, whatever makes sense for your data collection and testing system - even just falling back on random number generation, and writing that value out somewhere. The GWT I18n framework has several options built into it in its PropertyProviderGenerator implementation. You could even use this to read from a single meta tag, query param for all possible values, picking the char to read based on the order of all of the features that are being compiled in:
<meta name="ab-selection" content="acaababca">
would indicate 'a' for the first feature, 'c' for the second, etc.

Server participation:
Any server spitting out these meta tags will probably need to know what the possible options are - a generator (possibly in conjunction with a linker) can be set to spit out a file (a.k.a. artifact) with information collected during the compile. This could look something like a properties file, full of name=values pairs to define what the possible options are that need to be given to the running page. Presumably the client/server already have some way to discuss what is considered a 'successful' test, and to decide how to continue with future tests, but more metadata about those tests and how to run them could be added with additional annotations.

Useful examples, references:
http://code.google.com/p/google-web-toolkit/wiki/SoftPermutations - description of what soft perms are, how to use them
com.google.gwt.core.ext.Generator - base class for defining rebind rules in Java - can create new classes, or select from existing ones
com.google.gwt.core.ext.linker.PropertyProviderGenerator - interface that describes how to programmatically make permutation selection in the bootstrap
/com/google/gwt/user/UserAgent.gwt.xml - example of existing permutation selection wiring, and how to pass that off to a generator
com.google.gwt.user.rebind.UserAgentPropertyGenerator
/com/google/gwt/i18n/I18N.gwt.xml - more permutation selection, this time talking about reading from url, cookies, etc