Intelligence" <brain-training@googlegroups.com>
Received: by 10.43.84.4 with SMTP id ai4mr7029926icc.22.1353179883872;
        Sat, 17 Nov 2012 11:18:03 -0800 (PST)
X-BeenThere: brain-training@googlegroups.com
Received: by 10.50.36.133 with SMTP id q5ls1778578igj.29.canary; Sat, 17 Nov
 2012 11:18:00 -0800 (PST)
Received: by 10.68.251.136 with SMTP id zk8mr1969032pbc.1.1353179880118;
        Sat, 17 Nov 2012 11:18:00 -0800 (PST)
Received: by 10.68.251.136 with SMTP id zk8mr1969031pbc.1.1353179880109;
        Sat, 17 Nov 2012 11:18:00 -0800 (PST)
Return-Path: <jtoo...@jtoomim.org>
Received: from a.mail.sonic.net (a.mail.sonic.net. [64.142.16.245])
        by gmr-mx.google.com with ESMTPS id h8si819906pav.0.2012.11.17.11.17.59
        (version=TLSv1/SSLv3 cipher=OTHER);
        Sat, 17 Nov 2012 11:18:00 -0800 (PST)
Received-SPF: neutral (google.com: 64.142.16.245 is neither permitted nor denied by best guess record for domain of jtoo...@jtoomim.org) client-ip=64.142.16.245;
Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 64.142.16.245 is neither permitted nor denied by best guess record for domain of jtoo...@jtoomim.org) smtp.mail=jtoo...@jtoomim.org
Received: from [10.0.2.2] (c-67-171-158-99.hsd1.or.comcast.net [67.171.158.99])
	(authenticated bits=0)
	by a.mail.sonic.net (8.13.8.Beta0-Sonic/8.13.7) with ESMTP id qAHJHxj6007963
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <brain-training@googlegroups.com>; Sat, 17 Nov 2012 11:17:59 -0800
Message-ID: <50A7E2E6.1010304@jtoomim.org>
Date: Sat, 17 Nov 2012 11:17:58 -0800
From: Jonathan Toomim <jtoo...@jtoomim.org>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121026 Thunderbird/16.0.2
MIME-Version: 1.0
To: brain-training@googlegroups.com
Subject: Re: Meta-analysis: Preliminary training time results
References: <CAMwO0gzsiL-+UR-Vgxr2qhNKDaYbWUMHuj5_tk239Jp8o8Djtw@mail.gmail.com>
In-Reply-To: <CAMwO0gzsiL-+UR-Vgxr2qhNKDaYbWUMHuj5_tk239Jp8o8Djtw@mail.gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit

Thank you for doing this.

The results are disappointing. The slightly negative slope (i.e. groups 
improve more the less training they did) is not particularly consistent 
with an effect of DNB training, but consistent with publication bias 
(the less people invest in running a study, the more likely their 
decision to publish will be influenced by the results).

I'd like to link to some form of this analysis from the 
brainworkshop.net front page. Let me know if/when you write something up 
and post it on your website. A summary/abstract written for people who 
barely know what DNB and WM training are would be ideal.

Jonathan

On 11/16/2012 9:23 PM, Gwern Branwen wrote:
> I didn't feel like doing my other todos, so I cracked open the metafor
> (http://www.metafor-project.org/) documentation and it turned out to
> be easier than I expected to port my meta-analysis and then actually
> do some modeling. Compiling the training time was much more tedious
> and it looks like I need to unpool some entries or ask some people for
> more data to get a complete set. (For example, my Jaeggi 2008 entry
> pools all the training groups and so my training minutes total is also
> an average. Jaeggi gave me some data back in March, but it was pooled.
> Still, one swallow does not make a spring.) Plus I'm not entirely sure
> I'm doing it right since metafor is a powerful and general package.
>
> But with that in mind: there seems to be no relation.
>
> -----------------------------------
>
> The plain regression:
>
>      Mixed-Effects Model (k = 15; tau^2 estimator: REML)
>
>      tau^2 (estimate of residual amount of heterogeneity): 0.3392 (SE = 0.1833)
>      tau (sqrt of the estimate of residual heterogeneity): 0.5824
>
>      Test for Residual Heterogeneity:
>      QE(df = 13) = 51.0560, p-val < .0001
>
>      Test of Moderators (coefficient(s) 2):
>      QM(df = 1) = 0.4568, p-val = 0.4991
>
>     Model Results:
>
>               estimate      se     zval    pval    ci.lb   ci.ub
>      intrcpt    0.8047  0.3053   2.6354  0.0084   0.2062  1.4031  **
>      mods      -0.0003  0.0005  -0.6759  0.4991  -0.0012  0.0006
>
>      ---
>      Signif. codes:  0 �***� 0.001 �**� 0.01 �*� 0.05 �.� 0.1 � � 1
>
> There's no line which fits well and the estimate slope is slightly
> negative. That's not surprising when I graph effect size vs minutes of
> training:
>
> http://i.imgur.com/11qEw.png
>
> (The shorter studies seem to have a bit of a higher mean so when
> extrapolated all the way out to Jausvec's ridiculous amount of
> training, the best-fitting line - which is still badly fitting - goes
> downward a bit.)
>
> In contrast, the active/passive control group shows up as a powerful
> moderator even treated as a line thingy:
>
>      Mixed-Effects Model (k = 20; tau^2 estimator: REML)
>
>      tau^2 (estimate of residual amount of heterogeneity): 0.2322 (SE = 0.1231)
>      tau (sqrt of the estimate of residual heterogeneity): 0.4819
>
>      Test for Residual Heterogeneity:
>      QE(df = 18) = 50.8588, p-val < .0001
>
>      Test of Moderators (coefficient(s) 2):
>      QM(df = 1) = 4.8196, p-val = 0.0281
>
>      Model Results:
>
>               estimate      se     zval    pval    ci.lb    ci.ub
>      intrcpt    0.7861  0.1932   4.0694  <.0001   0.4075   1.1647  ***
>      mods      -0.6025  0.2744  -2.1954  0.0281  -1.1404  -0.0646    *
>
>      ---
>      Signif. codes:  0 �***� 0.001 �**� 0.01 �*� 0.05 �.� 0.1 � � 1
>
>
> There's a large slope which reaches significance (the negative is
> because the effect sizes shrinks as you go from passive/0 to
> active/1). This too is plausible if you've looked at my previous
> meta-analytic results, and is also quite obvious if we do a quick
> graph:
>
> http://i.imgur.com/qqrNi.png / http://i.imgur.com/aKJbq.png
>
> There's only 6 studies around or past 500 minutes, so the result isn't
> as good as it could be until I can incorporate Chooi 2010, unpool
> Jaeggi 2008, and get confirmation on whether my guess of 500 minutes
> for Jaeggi4 is right, but while I expect Jaeggi 2008 to show a trend
> (since that's what the original paper said) I don't expect much from
> Chooi 2010 or Jaeggi4, so probably the graph and model won't magically
> flip itself to show a line heading to the upper-right and vindicating
> all previous comments about training time determining benefits.
>