Meta-analysis: Preliminary training time results

[Gwern Branwen]

I didn't feel like doing my other todos, so I cracked open the metafor
(http://www.metafor-project.org/) documentation and it turned out to
be easier than I expected to port my meta-analysis and then actually
do some modeling. Compiling the training time was much more tedious
and it looks like I need to unpool some entries or ask some people for
more data to get a complete set. (For example, my Jaeggi 2008 entry
pools all the training groups and so my training minutes total is also
an average. Jaeggi gave me some data back in March, but it was pooled.
Still, one swallow does not make a spring.) Plus I'm not entirely sure
I'm doing it right since metafor is a powerful and general package.

But with that in mind: there seems to be no relation.

-----------------------------------

The plain regression:

Mixed-Effects Model (k = 15; tau^2 estimator: REML)

tau^2 (estimate of residual amount of heterogeneity): 0.3392 (SE = 0.1833)
tau (sqrt of the estimate of residual heterogeneity): 0.5824

Test for Residual Heterogeneity:
QE(df = 13) = 51.0560, p-val < .0001

Test of Moderators (coefficient(s) 2):
QM(df = 1) = 0.4568, p-val = 0.4991

Model Results:

estimate se zval pval ci.lb ci.ub
intrcpt 0.8047 0.3053 2.6354 0.0084 0.2062 1.4031 **
mods -0.0003 0.0005 -0.6759 0.4991 -0.0012 0.0006

---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

There's no line which fits well and the estimate slope is slightly
negative. That's not surprising when I graph effect size vs minutes of
training:

http://i.imgur.com/11qEw.png

(The shorter studies seem to have a bit of a higher mean so when
extrapolated all the way out to Jausvec's ridiculous amount of
training, the best-fitting line - which is still badly fitting - goes
downward a bit.)

In contrast, the active/passive control group shows up as a powerful
moderator even treated as a line thingy:

Mixed-Effects Model (k = 20; tau^2 estimator: REML)

tau^2 (estimate of residual amount of heterogeneity): 0.2322 (SE = 0.1231)
tau (sqrt of the estimate of residual heterogeneity): 0.4819

Test for Residual Heterogeneity:
QE(df = 18) = 50.8588, p-val < .0001

Test of Moderators (coefficient(s) 2):
QM(df = 1) = 4.8196, p-val = 0.0281

Model Results:

estimate se zval pval ci.lb ci.ub
intrcpt 0.7861 0.1932 4.0694 <.0001 0.4075 1.1647 ***
mods -0.6025 0.2744 -2.1954 0.0281 -1.1404 -0.0646 *

---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


There's a large slope which reaches significance (the negative is
because the effect sizes shrinks as you go from passive/0 to
active/1). This too is plausible if you've looked at my previous
meta-analytic results, and is also quite obvious if we do a quick
graph:

http://i.imgur.com/qqrNi.png / http://i.imgur.com/aKJbq.png

There's only 6 studies around or past 500 minutes, so the result isn't
as good as it could be until I can incorporate Chooi 2010, unpool
Jaeggi 2008, and get confirmation on whether my guess of 500 minutes
for Jaeggi4 is right, but while I expect Jaeggi 2008 to show a trend
(since that's what the original paper said) I don't expect much from
Chooi 2010 or Jaeggi4, so probably the graph and model won't magically
flip itself to show a line heading to the upper-right and vindicating
all previous comments about training time determining benefits.

--
gwern
http://www.gwern.net/DNB%20FAQ#meta-analysis

[Payman Saghafi]

Great work, Gwern.  Good effort.

By the way, I feel sorry for that swallow; it can't make a spring or summer.  :p

Pay

[Jonathan Toomim]

Thank you for doing this.

The results are disappointing. The slightly negative slope (i.e. groups
improve more the less training they did) is not particularly consistent
with an effect of DNB training, but consistent with publication bias
(the less people invest in running a study, the more likely their
decision to publish will be influenced by the results).

I'd like to link to some form of this analysis from the
brainworkshop.net front page. Let me know if/when you write something up
and post it on your website. A summary/abstract written for people who
barely know what DNB and WM training are would be ideal.

Jonathan

> Signif. codes: 0 �***� 0.001 �**� 0.01 �*� 0.05 �.� 0.1 � � 1

>
> There's no line which fits well and the estimate slope is slightly
> negative. That's not surprising when I graph effect size vs minutes of
> training:
>
> http://i.imgur.com/11qEw.png
>
> (The shorter studies seem to have a bit of a higher mean so when
> extrapolated all the way out to Jausvec's ridiculous amount of
> training, the best-fitting line - which is still badly fitting - goes
> downward a bit.)
>
> In contrast, the active/passive control group shows up as a powerful
> moderator even treated as a line thingy:
>
> Mixed-Effects Model (k = 20; tau^2 estimator: REML)
>
> tau^2 (estimate of residual amount of heterogeneity): 0.2322 (SE = 0.1231)
> tau (sqrt of the estimate of residual heterogeneity): 0.4819
>
> Test for Residual Heterogeneity:
> QE(df = 18) = 50.8588, p-val < .0001
>
> Test of Moderators (coefficient(s) 2):
> QM(df = 1) = 4.8196, p-val = 0.0281
>
> Model Results:
>
> estimate se zval pval ci.lb ci.ub
> intrcpt 0.7861 0.1932 4.0694 <.0001 0.4075 1.1647 ***
> mods -0.6025 0.2744 -2.1954 0.0281 -1.1404 -0.0646 *
>
> ---

> Signif. codes: 0 �***� 0.001 �**� 0.01 �*� 0.05 �.� 0.1 � � 1

[Zaraki]

Yes, thank you.

Looks like another hole got plugged with this - few remains. I would still like to know what happens to make some start to have lucid dreaming or improved dream recall. Does anyone know if there is any research on this connected to meditation?

[jttoto2]

Ever since I picked up meditation I noticed recalling my dreams nearly every day.  If by lucid dreaming you mean, "the ability to control one's dreams", then I haven't experienced this personally.  I am in a hurry and don't have time to find the study, but there was a paper showing that people who practice meditation experience more REM sleep.  You can find this with a quick google scholar search.  More REM doesn't automatically mean deeper REM, but I'm open to the possibility that there is a connection.  

[Gwern Branwen]

On Sat, Nov 17, 2012 at 2:17 PM, Jonathan Toomim <jto...@jtoomim.org> wrote:
> The results are disappointing. The slightly negative slope (i.e. groups
> improve more the less training they did) is not particularly consistent with
> an effect of DNB training, but consistent with publication bias (the less
> people invest in running a study, the more likely their decision to publish
> will be influenced by the results).

I take it as being mostly consistent with minimal dose-response, or
perhaps an initial benefit followed by task-specific training.

Publication bias-wise, remember that the graph isn't weighted by
sample size in any way, so tiny studies look the same as large
studies; I've incorporated the usual publication bias checks into my
meta-analysis pretty much from the start:
http://www.gwern.net/DNB%20FAQ#biases

Both the funnel plots and trim-and-fills don't detect any real sign of
publication bias. I'd say the more accurate description of the
situation is that there is a '*publicity* bias', where my
comprehensive collection of n-back & IQ studies is unbiased but the
distribution of citations is heavily unbalanced towards the positive
results (although I haven't looked into citation statistics; maybe one
day).

> I'd like to link to some form of this analysis from the brainworkshop.net
> front page. Let me know if/when you write something up and post it on your
> website. A summary/abstract written for people who barely know what DNB and
> WM training are would be ideal.

I'm not sure how important my meta-analysis should be considered,
since strictly speaking I'm an amateur and this was also my first
meta-analysis; plus I know it's incomplete in several ways... I may
never feel comfortable endorsing a summary to splash across the BW
homepage especially since it's probably going to be a fairly negative
sort of summary which would put off many people.

On Sat, Nov 17, 2012 at 5:57 PM, jttoto2 <john....@gmail.com> wrote:
> I am in a hurry and don't have time to find the study, but there was a paper
> showing that people who practice meditation experience more REM sleep. You
> can find this with a quick google scholar search.

Are you referring to "Meditation acutely improves psychomotor
vigilance, and may decrease sleep need"
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2919439/ Kaul et al 2010?
Well, it's possible. Hopefully my big sleep experiment, which includes
meditation as an intervention, will say something interesting whenever
I finish it.

--
gwern
http://www.gwern.net/Zeo

[jttoto2]

I can't seem to find the study at the moment but here is a recent link showing that meditators show enhanced REM states.  This is more pertinent than the one I was thinking of.  

Link:  http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3328970/

[Gwern Branwen]

I've finished up some of the loose ends and wrote up the new version
properly, much as before: http://www.gwern.net/DNB%20FAQ#analysis

No modification to my previous conclusions.

(I'm afraid it's not quite as pretty as it was before; I don't
understand the graphics options in `metafor` too well.)

--
gwern