Meta regression on payment in n-back studies, Re: Video with Jaeggi on current debates within brain training

[Gwern Branwen]

re http://vimeo.com/65064957

On Wed, May 15, 2013 at 12:41 AM, ☉ <argu...@gmail.com> wrote:
> Yeah, I had an hour of free time, so here's basically a copy of the
> slideshow with notes of various things said during the video. It
> sounds like she's coming around to realizing that n-back training is
> likely best for kids with ADHD over all (what I've pretty much been
> saying), where those with ADHD are apparently better situated to
> benefit from training on fluid intelligence measures due to training's
> apparent influence in improving how well such persons can handle
> "interference" (sorting pink noise from the white noise, maybe?).

Thank you for the summary.

On Thu, May 16, 2013 at 2:38 PM, Geno <ma...@alejandrolc.com> wrote:
> She addresses some of the studies that have shown no significant gains and
> suggests that the results could be attributed to an extrinsic motivator
> (money). In other words, it seems that studies that provide little to no
> economic incentive have shown the best results for transfer.

This seems unlikely to be any significant factor. I spent the last
hour or so coding up payment in my meta-analysis (same place as
always, http://www.gwern.net/DNB%20FAQ#meta-analysis ) and then
testing it. I have cc'd Jaeggi as no doubt the results will be of
interest to her.

----

Payment seems to actually be quite rare in n-back studies (in part
because it's so common to just recruit students with course credit or
extra credit), and so as a moderator it is currently a small and
non-statistically-significant negative effect, whether you regress on
the total payment amount or treat it as a boolean variable. More
interestingly, it seems that the negative sign is being driven by
payment being associated with higher-quality studies using active
control groups, because when you look at the interaction, payment in a
study with an active control group actually flips sign to being
positive again (correlating with a bigger effect size).

More specifically. If we check payment as a binary variable, we get a
decrease in effect size which is not statistically-significant:

Mixed-Effects Model (k = 43; tau^2 estimator: REML)
...
Test of Moderators (coefficient(s) 2):
QM(df = 1) = 0.8511, p-val = 0.3563

Model Results:

estimate se zval pval ci.lb ci.ub
intrcpt 0.5701 0.1076 5.2978 <.0001 0.3592 0.7810
as.logical(Paid)TRUE -0.2139 0.2319 -0.9225 0.3563 -0.6684 0.2406

If we instead regress against the total payment size (perhaps larger
payments discourage one more?), the effect of each additional dollar
is very small and 0 is far from excluded as the coefficient:

Test of Moderators (coefficient(s) 2):
QM(df = 1) = 0.9653, p-val = 0.3259

Model Results:

estimate se zval pval ci.lb ci.ub
intrcpt 0.5618 0.1028 5.4675 <.0001 0.3604 0.7632
Paid -0.0017 0.0018 -0.9825 0.3259 -0.0052 0.0017

Finally, as I've mentioned before, the difference in effect size
between active and passive control groups is quite striking, and I
noticed that the Redick et al 2012 experiment paid subjects a lot of
money to put up with all its tests and ensure subject retention, so
what happens if we look for an interaction?

Test of Moderators (coefficient(s) 2,3,4):
QM(df = 3) = 6.7326, p-val = 0.0809

Model Results:

estimate se zval pval
ci.lb ci.ub
intrcpt 0.7307 0.1256 5.8188 <.0001
0.4846 0.9768
active -0.4810 0.2146 -2.2416 0.0250
-0.9016 -0.0604
as.logical(Paid)TRUE -0.2722 0.2589 -1.0513 0.2931
-0.7796 0.2352
active:as.logical(Paid)TRUE 0.1054 0.4805 0.2193 0.8264
-0.8364 1.0472

Active control groups cut the observed effect of n-back by more than
half, as usual, and passive+payment shrinks the effect size by ~34%,
but active+payment correlates with an increased effect by 16%.

Hence, I infer that the previous negative correlation is probably
because all the passive experiments are also not paying (no need to
pay controls if they're not doing anything besides showing up twice
for testing, really), but the true impact of payment is, if anything,
probably opposite of Jaeggi's suggestion.

--
gwern