about Overlapping Experiment

[Hongyu Bi]

Hi ,

Does proctor support overlapping experiment like this : 

http://research.google.com/pubs/pub36500.html 

thanks:)

[dmcm...@indeed.com]

Hongyu,

Apologies for the slow response to your question.

The short answer is yes, proctor does indeed support overlapping experiments. Each newly created experiment randomly assigns units/users/cookies across a fresh over-lapping layer. This is achieved via a new salt under the hood. Details are contained in this tech blog and here is the full documentation. 

However, proctor does not support some of the other features outlined in the Google paper:

• Long-lasting experiment layers, to ensure that experiments don't interact in destructive manners, e.g. the referenced UI layers to avoid pink text on pink background
• No differentiation between launch and experiment layers.
• Re-randomization of units within the experiment layer. 

Note that all of these could be recreated in a simple manner using proctor and the retention of the associated salt.

Please let me know if this does not answer your question completely,

Donal

[Amit Gelber]

Hi Donal, 

Can you please elaborate on 

"Note that all of these could be recreated in a simple manner using proctor and the retention of the associated salt."

My goal is to ensure that user with id 100 will be allocated in test 1 to group X

and later on in test 2, the same user 100 will be allocated to control since test 1 and test 2 are correlated.

how can I achieve that?

Thanks.

[matts]

Hi, Amit!

If you take a look at the JSON Schema for the test definition...

http://opensource.indeedeng.io/proctor/docs/test-definition/

... you'll find the "salt" parameter:

A salt in the hashing function used to map String identifiers to integer values. A good convention is to use the test name. Salts that start with “&” allow you to align bucket assignments by using identical salts.

If you need to ensure that two tests have correlated membership, you can give them the same salt string, which must start with an ampersand. (This is intentionally a little hard to do accidentally, since you usually don't want your tests to be correlated).

That setup will cause the identifier to hash to the same numeric value for test bucket assignment. If the two tests also use the same allocation ranges, then you can deterministically guarantee that the same user will receive the test treatment from one experiment and the control treatment from the other.

[Amit Gelber]

Thank you so much for the quick response!

I check that out.

Is this subject addressed in proctor web-app?

[matts]

Yes, Proctor webapp will allow you to set the salt value for a new test. 

I don't believe there is any built-in help-text around the special-case for sharing salts, beyond the system documentation I linked below. That could make a good inline hint as well.

[Amit Gelber]

So I'm testing the & feature(correlated tests):

• My test matrix is as follows:

{
"tests": {
"testaichorn": {
"description": "testing if the AI model is better than old model",
"version": 1,
"rule": null,
"constants": {
},
"salt": "testaichorn",
"buckets": [
{
"name": "inactive",
"value": -1,
"description": "the inactive case",
"payload": {
"stringValue": "some val 1"
}
},
{
"name": "control",
"value": 0,
"description": "the control case (stays old model)",
"payload": {
"stringValue": "some val 1"
}
},
{
"name": "AI model",
"value": 1,
"description": "AI model, trained by someone at date ddmmyy",
"payload": {
"stringValue": "some val 2"
}
},
{
"name": "some other model",
"value": 2,
"description": "neon green to really get the user's attention",
"payload": {
"stringValue": "some val 3"
}
}
],
"allocations": [
{
"rule": null,
"ranges": [
{
"length": 0.0,
"bucketValue": -1
},
{
"length": 0.3,
"bucketValue": 1
},
{
"length": 0.4,
"bucketValue": 0
},
{
"length": 0.3,
"bucketValue": 2
}
]
}
],
"testType": "USER"
},
"some test 1": {
"description": "this test is correlated with test 2",
"version": 1,
"rule": null,
"constants": {
},
"salt": "&correlatedtest",
"buckets": [
{
"name": "inactive",
"value": -1,
"description": "the inactive case",
"payload": {
"stringValue": "some val 1"
}
},
{
"name": "control",
"value": 0,
"description": "the control case (stays old model)",
"payload": {
"stringValue": "some val 1"
}
},
{
"name": "AI model",
"value": 1,
"description": "AI model, trained by someone at date ddmmyy",
"payload": {
"stringValue": "some val 2"
}
},
{
"name": "some other model",
"value": 2,
"description": "neon green to really get the user's attention",
"payload": {
"stringValue": "some val 3"
}
}
],
"allocations": [
{
"rule": null,
"ranges": [
{
"length": 0.0,
"bucketValue": -1
},
{
"length": 0.3,
"bucketValue": 1
},
{
"length": 0.4,
"bucketValue": 0
},
{
"length": 0.3,
"bucketValue": 2
}
]
}
],
"testType": "USER"
},
"some test 2": {
"description": "this test is correlated with test 1",
"version": 1,
"rule": null,
"constants": {
},
"salt": "&correlatedtest",
"buckets": [
{
"name": "inactive",
"value": -1,
"description": "the inactive case",
"payload": {
"stringValue": "some val 1"
}
},
{
"name": "control",
"value": 0,
"description": "the control case (stays old model)",
"payload": {
"stringValue": "some val 1"
}
},
{
"name": "AI model",
"value": 1,
"description": "AI model, trained by someone at date ddmmyy",
"payload": {
"stringValue": "some val 2"
}
},
{
"name": "some other model",
"value": 2,
"description": "neon green to really get the user's attention",
"payload": {
"stringValue": "some val 3"
}
}
],
"allocations": [
{
"rule": null,
"ranges": [
{
"length": 0.0,
"bucketValue": -1
},
{
"length": 0.3,
"bucketValue": 1
},
{
"length": 0.4,
"bucketValue": 0
},
{
"length": 0.3,
"bucketValue": 2
}
]
}
],
"testType": "USER"
}

},
"audit": {
"version": 4,
"updatedBy": "Amit Gelber",
"updated": 1
}
}

• the REST call is http://localhost:8080/groups/identify?ctx.country=US&ctx.loggedIn=false&id.USER=2 ( using Pipet)
• answer is:

{ "data" : { "groups" : { "testaichorn" : { "name" : "AI model", "value" : 1, "payload" : { "stringValue" : "some val 2" }, "version" : "1" }, "some test 1" : { "name" : "AI model", "value" : 1, "payload" : { "stringValue" : "some val 2" }, "version" : "1" }, "some test 2" : { "name" : "AI model", "value" : 1, "payload" : { "stringValue" : "some val 2" }, "version" : "1" } }, "context" : { "country" : "US", "loggedIn" : false, "userAgent" : { "os" : { "version" : "", "majorVersion" : -1, "minorVersion" : -1, "family" : "mac_os_x" }, "version" : { "version" : "87.0.4280.67", "major" : 87, "minor" : 0 }, "userAgentString" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36", "android" : false, "browser" : "CHROME8", "browserVersion" : { "version" : "87.0.4280.67", "majorVersion" : "87", "minorVersion" : "0" }, "operatingSystem" : "MAC_OS_X", "deviceType" : "COMPUTER", "tablet" : false, "ios" : false, "chromeForIOS" : false, "windowsPhone" : false, "mobileDevice" : false, "smartPhone" : false, "phone" : false, "dumbPhone" : false } }, "audit" : { "version" : "4", "updated" : 1, "updatedDate" : "1969-12-31T18:00-0600", "updatedBy" : "Amit Gelber" } }, "meta" : { "status" : 200 } }

• Some test 1 supposed to be correlated with some test 2.
• I used for both tests "&correlatedtest" at the salt field.
• Expected: in test 1, user id 2 gets treatment, in test 2, user id  2 gets control( or the other way around).
• What I got: in both cases user id 2 got treatment( AI model )

[matts]

From the details you paste, it appears things are operating correctly.

When two tests share a salt, they'll share a user distribution.

Your allocation for "Some test 1" is [bucket(1) @ 0.3, bucket(0) @ 0.4, bucket (2) @ 0.3]. Your allocation for "Some test 2" is the same.  So these tests are 100% correlated, as you observed: any user that ends up in bucket 0, 1, or 2 in the first test will end up in the same bucket in the other.

If you want to anti-correlate the bucket assignment, so that a given user always gets the control (or rather, "inactive") group in one test and is split between the control groups in the other, you'd do something like:

Test 1: [ inactive @ 0.5,  aiModel @ 0.16, control @ 0.17, otherModel @ 0.16]

Test 2: [ aiModel @ 0.16, control @ 0.17, otherModel @ 0.16, inactive @ 0.5]

That way, you ensure that the users that view Test 1's test treatments never see Test 2's test treatments. (Of course, this halves the number of users that receive the treatments, so you'll need to consider that when doing your power analysis).

Did I understand your goal properly?

[Amit]

Yes, you are right. I'm looking for anti-correlated tests.

I'm still not sure what "&" is used for. can't I just use the same salt key?

Another question is, in your example, you moved the groups one step to the right.

 To my understanding, the only thing it does is to make sure the same user won't get the same allocation as in test 1 and in test 2, am I right?

BUT, you still can't force the allocation to be a specific group? for instance, treatment in test 1, and force control in test 2?

Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "indeedeng-proctor-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/indeedeng-proctor-users/PkaDBpY4FPc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to indeedeng-proctor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/indeedeng-proctor-users/53f50cf0-483f-49e4-9bb9-c99a72a1e9bdn%40googlegroups.com.

[matts]

can't I just use the same salt key?

That's a terribly reasonable question :) In the early days of Proctor, the system worked hard to prevent users from accidentally correlating test buckets, so just using the same salt string would not result in correlated tests. We eventually realized that we needed the ability to force correlation in some cases and introduced the "&" prefix convention as a way to indicate to the system that "yes, we really intend these distributions to be correlated". Admittedly, a bit opaque.

in your example

In my example, I made sure that the control and both treatment groups lined up in each correlated test lined up with an inactive group in the other. The inactive and control treatments should be identical in the user experience, the system simply does log control assignments and doesn't log inactive ones. This keeps the population size of the "control" and each test treatment nearly equal, even though 2/3 (1/2 + 1/6) of users are receiving the baseline experience. That equality just simplifies the test analysis.

 

It's easiest to think of the allocation range as a number line. In my example allocation ranges, the first half of the number line is "inactive" for test1 and control/model1/model2 for test2; the second half of the number line is the opposite. Every individual user ID hashes to the same specific value on both number lines, so that user is "inactive" in one test and assigned to control, model1, or model2 in the other.

I tweaked the positions of the logged treatments to put control in the middle, but in retrospect, that's just habit and totally unnecessary here. When you're running simple A/B tests, you'll usually want to put the inactive group in the middle rather than on one end or the other, so that you can grow the control and test groups over time without causing reassignments. It's common to start A/B tests at very low percentages until you're sure the test treatments aren't super-negative for some reason, then slowly grow the control and test groups to get more data.  That's not at all the point of this exercise, though, so was needlessly confusing.

[Amit Gelber]

Hi,

got your point regarding the "&" and number lines.

another followup question:

What happens if there is 3 anti-correlated tests? and 4?

how do I manipulate the config for something like https://facebook.github.io/planout/docs/namespaces.html?

Thanks again!

[matts]

If you want a single user to be assigned to at most one experiment from a set, then the easiest thing to do is to create a single proctor test and use that as the namespace. Within the same number line, you can create a control and test bucket for each experiment you're running, being sure that any given user will be assigned to at most one experiment.

You need to be cautious managing that sort of test, of course, for several reasons. Finely subdividing your population has a big effect on the experiment's power and time to significance. You lose the ability to easily "grow" your control and test buckets from 1% -> 5% -> 10%, etc, to achieve results more quickly. Unless you're looking at massive levels of traffic, you might consider running truly incompatible experiments sequentially; maybe even following the Amazon pattern of strictly (procedurally) limiting the number of experiments that affect a given area of the presentation.

It's important to note: these limitations apply regardless of the implementation method. Namespaces, line numbers, universes, etc... these are all just different semantic glosses on top of dividing up the user space in a collection of non-overlapping buckets. The key numbers you need to worry about are the number of exposures and the effect size you're looking to detect.

[Amit Gelber]

"then the easiest thing to do is to create a single proctor test and use that as the namespace"

How do I do this?

[matts]

Let's say you wanted to have the ability to run 10 concurrent, non-overlapping experiments on a search results page.

You could set up a test named "searchResultsTests" with buckets like:

- control0 (5%)

- test0 (5%)

- control1 (5%)

- test1 (5%)

...

- control9 (5%)

- test9 (5%)

You would then implement each experiment in one of the testN buckets, being sure that no user could belong to multiple testN buckets.