Redick et al 2012 interview
138 views
Skip to first unread message
Gwern Branwen
May 30, 2012, 9:36:06 AM5/30/12
to N-back
From Google Alerts:
http://www.smartplanet.com/blog/thinking-tech/q-a-new-evidence-shows-brain-training-games-dont-work/11758
"Q&A: New evidence shows brain-training games don’t work"
> ...In fact, David Z. Hambrick, associate professor of psychology at Michigan State University, and his colleagues Thomas S. Redick (lead researcher) and Randall W. Engle will soon be publishing new evidence that fails to replicate the very study that so much of the commercial industry rests upon.
> ...SP: What was the trigger that launched this huge industry trend of brain games?
>
> DZH: Psychologists have been interested in the idea of improving intelligence for over a century. But until the mid-2000s, people were not very optimistic about this whole enterprise.
>
> However, there was a specific study published in 2008 in the Proceedings of the National Academy of Sciences by Susanne Jaeggi and Martin Buschkuehl that renewed interest in this topic.
> ...SP: But you claim the study has major flaws?
>
> DZH: If you find that people get better in one test of reasoning it doesn’t mean necessarily that they’re smart, it means that they’re better on one test of reasoning. You can’t measure fluid intelligence with any single test, it’s measured with multiple tests.
>
> SP: And the other flaw?
>
> DZH: There were some pretty striking differences between the control group and the training groups. The control group who received no training, went home and did whatever. But the training groups, on the other hand, came in regularly for training. This raises possibility of motivation being an explanation: They wanted to do well in the experiment.
>
> Another important point is that there were procedural differences across these training groups that really complicate interpretation of the results, and in particular the claim that more training equals more gain. These procedural differences were not reported in the Jaeggi article. We found out about them in Jaeggi’s unpublished dissertation, and through follow-up emails to Jaeggi.
>
> SP: You worked with lead researcher Tom Redick to attempt to replicate the findings from the Jaeggi study. And this research is about to be published. Can you give us a preview of what you found?
>
> DZH: So, we set out to replicate the findings, correcting all of these problems. We had subjects complete not one but eight tests of fluid intelligence. We then assigned them to a training group in which they received 20 sessions of training in Jaeggi’s dual n-back task or to one of two control conditions. The “no-contact” control condition was the same as Jaeggi’s control condition. By contrast, in the “active-control” condition, subjects were trained in a task that we designed to be as demanding as dual n-back without tapping working memory capacity. Finally, we had all subjects perform different versions of the eight intelligence tests half way through training and at the end.
>
> And what did we find? Zip. There wasn’t much more than a hint of the pattern of results that Jaeggi reported in any of the eight intelligence tests, and nothing in the predicted direction that even approached statistical significance. If you someone were to ask me to estimate how much 20 sessions of training in dual n-back tasks improves fluid intelligence, I’d say zero.
> ...SP: Because it’s a value that society deems important, and it has very real implications, right?
>
> DZH: Well one thing we know for sure is that it’s practically useful. IQ, despite what people will say about being a meaningless number, predicts a lot of things. It predicts job performance better than any single variable that we know of. It predicts educational attainment, it predicts income, it predicts health, it predicts longevity even after you take into account sort of obvious confounding factors like socio-economic status. So it captures something important and something useful. If we were to no longer use IQ tests, for things like personnel tests of college admissions, it would cost society billions. IQ is the single best predictor of a lot of outcomes that we value in society.
--
gwern
http://www.gwern.net
http://www.smartplanet.com/blog/thinking-tech/q-a-new-evidence-shows-brain-training-games-dont-work/11758
"Q&A: New evidence shows brain-training games don’t work"
> ...In fact, David Z. Hambrick, associate professor of psychology at Michigan State University, and his colleagues Thomas S. Redick (lead researcher) and Randall W. Engle will soon be publishing new evidence that fails to replicate the very study that so much of the commercial industry rests upon.
> ...SP: What was the trigger that launched this huge industry trend of brain games?
>
> DZH: Psychologists have been interested in the idea of improving intelligence for over a century. But until the mid-2000s, people were not very optimistic about this whole enterprise.
>
> However, there was a specific study published in 2008 in the Proceedings of the National Academy of Sciences by Susanne Jaeggi and Martin Buschkuehl that renewed interest in this topic.
> ...SP: But you claim the study has major flaws?
>
> DZH: If you find that people get better in one test of reasoning it doesn’t mean necessarily that they’re smart, it means that they’re better on one test of reasoning. You can’t measure fluid intelligence with any single test, it’s measured with multiple tests.
>
> SP: And the other flaw?
>
> DZH: There were some pretty striking differences between the control group and the training groups. The control group who received no training, went home and did whatever. But the training groups, on the other hand, came in regularly for training. This raises possibility of motivation being an explanation: They wanted to do well in the experiment.
>
> Another important point is that there were procedural differences across these training groups that really complicate interpretation of the results, and in particular the claim that more training equals more gain. These procedural differences were not reported in the Jaeggi article. We found out about them in Jaeggi’s unpublished dissertation, and through follow-up emails to Jaeggi.
>
> SP: You worked with lead researcher Tom Redick to attempt to replicate the findings from the Jaeggi study. And this research is about to be published. Can you give us a preview of what you found?
>
> DZH: So, we set out to replicate the findings, correcting all of these problems. We had subjects complete not one but eight tests of fluid intelligence. We then assigned them to a training group in which they received 20 sessions of training in Jaeggi’s dual n-back task or to one of two control conditions. The “no-contact” control condition was the same as Jaeggi’s control condition. By contrast, in the “active-control” condition, subjects were trained in a task that we designed to be as demanding as dual n-back without tapping working memory capacity. Finally, we had all subjects perform different versions of the eight intelligence tests half way through training and at the end.
>
> And what did we find? Zip. There wasn’t much more than a hint of the pattern of results that Jaeggi reported in any of the eight intelligence tests, and nothing in the predicted direction that even approached statistical significance. If you someone were to ask me to estimate how much 20 sessions of training in dual n-back tasks improves fluid intelligence, I’d say zero.
> ...SP: Because it’s a value that society deems important, and it has very real implications, right?
>
> DZH: Well one thing we know for sure is that it’s practically useful. IQ, despite what people will say about being a meaningless number, predicts a lot of things. It predicts job performance better than any single variable that we know of. It predicts educational attainment, it predicts income, it predicts health, it predicts longevity even after you take into account sort of obvious confounding factors like socio-economic status. So it captures something important and something useful. If we were to no longer use IQ tests, for things like personnel tests of college admissions, it would cost society billions. IQ is the single best predictor of a lot of outcomes that we value in society.
--
gwern
http://www.gwern.net
J.
May 30, 2012, 10:51:43 AM5/30/12
to brain-t...@googlegroups.com
Am I reading this correctly, or did they only do 20 sessions of DNB? If he's using the term session the same way that we are, that's a single day of training. Maybe he means 20 days of 20 sessions?
Gwern Branwen
May 30, 2012, 11:05:54 AM5/30/12
to brain-t...@googlegroups.com
On Wed, May 30, 2012 at 10:51 AM, J. <joseph.a.a...@gmail.com> wrote:
> Maybe he means 20 days of 20 sessions?
He means it however Jaeggi 2008 meant it.
> Maybe he means 20 days of 20 sessions?
--
gwern
http://www.gwern.net
Message has been deleted
XFMQ902SF
May 30, 2012, 11:36:14 AM5/30/12
to Dual N-Back, Brain Training & Intelligence
Gwern said 5-6 weeks is the estimated wait for Redick 2012 to be
published.
On May 30, 11:32 am, "J." <joseph.a.albrecht...@gmail.com> wrote:
> It looks to me like jaeggi 2008 used the term "session" to refer to a days
> worth of training (20 sessions by this groups terminology).
>
> I'd say this is a major damper on my enthusiasm for the DNB task. At least
> someone is finally crossing their 't's and dotting their 'i's.
>
> Seeing as they did testing at 3 time points and that there were apparently
> 'hints of the pattern', I'm curious to see whether or not the p values
> shifted at all between the 3 time points.
>
> I'm also curious about the number of subjects in each condition.
>
> I'm aware that the next question is beating on DNBs chest in my attempts to
> resuscitate, but do you think that 10 days of training may be too early to
> detect differences in intelligence and that the testing afforded the
> control and active-control groups an opportunity for practice on the tests
> that could have masked the cumulative effects on day 20?
>
> Any idea as to when this will be published?
published.
On May 30, 11:32 am, "J." <joseph.a.albrecht...@gmail.com> wrote:
> It looks to me like jaeggi 2008 used the term "session" to refer to a days
> worth of training (20 sessions by this groups terminology).
>
> I'd say this is a major damper on my enthusiasm for the DNB task. At least
> someone is finally crossing their 't's and dotting their 'i's.
>
> Seeing as they did testing at 3 time points and that there were apparently
> 'hints of the pattern', I'm curious to see whether or not the p values
> shifted at all between the 3 time points.
>
> I'm also curious about the number of subjects in each condition.
>
> I'm aware that the next question is beating on DNBs chest in my attempts to
> resuscitate, but do you think that 10 days of training may be too early to
> detect differences in intelligence and that the testing afforded the
> control and active-control groups an opportunity for practice on the tests
> that could have masked the cumulative effects on day 20?
>
> Any idea as to when this will be published?
Gwern Branwen
May 30, 2012, 11:46:26 AM5/30/12
to brain-t...@googlegroups.com
On Wed, May 30, 2012 at 11:32 AM, J. <joseph.a.a...@gmail.com> wrote:
> I'm also curious about the number of subjects in each condition.
http://www.gwern.net/DNB%20FAQ#fn78
control: n=20 6 (3); search: n=29 6.24 (3.34); n-back: n=24 6.25 (3.08)
--
gwern
http://www.gwern.net
polar
May 31, 2012, 8:10:08 AM5/31/12
to brain-t...@googlegroups.com
Thanks Gwern, this one sounds interesting and important, as they are WM veteran researchers. Nevertheless, there are some things worth mentioning here from my notes. One is, that they operate with working memory *capacity* (as opposed to say, cognitive process of updating). Ok, but WMC was repeteadly found to correlate higly with Gf, even by Engle alone (1999), Suss 2001, Colom 2003, Conway 2003, or Fukuda (2010) (e.g. quoting "...results suggest that the relationship between WM capacity and fluid intelligence is driven solely by the ability to hold multiple discrete representations in WM and not by the clarity of those representations.") On the other hand, Conway (2007) stated that n-back and ospan (WMC test) are independently responsible for variations in Gf. So how do they explain today, that improving WMC does not transfer to intelligence? There is a possibility N-back does not improve WM quantity, but quality (its effect is claimed to be independent from reached N, altough N correlates with actual Gf).
But of course, as they simply repeated Jaeggi (2008), so they should get similar results. But the second thing is, that giving somedy 8 or 16 iq tests before the final one (pardon, final 8), can pretty much equalize some five points of iq (potentially gained from the training) across the control groups. Lets wait for the paper.
Palguay
May 31, 2012, 8:25:52 AM5/31/12
to brain-t...@googlegroups.com
I had sent this earlier but seems like it did not get through, here is a link to the paper
http://scottbarrykaufman.com/article/study-alert-no-evidence-of-intelligence-improvement-after-working-memory-training-a-randomized-placebo-controlled-study/
http://scottbarrykaufman.com/article/study-alert-no-evidence-of-intelligence-improvement-after-working-memory-training-a-randomized-placebo-controlled-study/
On Thu, May 31, 2012 at 5:40 PM, polar <pol...@gmail.com> wrote:
Thanks Gwern, this one sounds interesting and important, as they are WM veteran researchers. Nevertheless, there are some things worth mentioning here from my notes. One is, that they operate with working memory *capacity* (as opposed to say, cognitive process of updating). Ok, but WMC was repeteadly found to correlate higly with Gf, even by Engle alone (1999), Suss 2001, Colom 2003, Conway 2003, or Fukuda (2010) (e.g. quoting "...results suggest that the relationship between WM capacity and fluid intelligence is driven solely by the ability to hold multiple discrete representations in WM and not by the clarity of those representations.") On the other hand, Conway (2007) stated that n-back and ospan (WMC test) are independently responsible for variations in Gf. So how do they explain today, that improving WMC does not transfer to intelligence? There is a possibility N-back does not improve WM quantity, but quality (its effect is claimed to be independent from reached N, altough N correlates with actual Gf).
But of course, as they simply repeated Jaeggi (2008), so they should get similar results. But the second thing is, that giving somedy 8 or 16 iq tests before the final one (pardon, final 8), can pretty much equalize some five points of iq (potentially gained from the training) across the control groups. Lets wait for the paper.
--To view this discussion on the web visit https://groups.google.com/d/msg/brain-training/-/p2_BZ_DAiacJ.
You received this message because you are subscribed to the Google Groups "Dual N-Back, Brain Training & Intelligence" group.
To post to this group, send email to brain-t...@googlegroups.com.
To unsubscribe from this group, send email to brain-trainin...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/brain-training?hl=en.
Reply all
Reply to author
Forward
0 new messages