Summary: does anyone know how I should go about this? Are there
any Michigan residents who might be interested in helping?

----------

So some background first: as long-time readers probably know, as I continued
compiling n-back studies for the FAQ, I became increasingly skeptical that the
results were being driven by the methodological shortcuts & sloppiness
identified by
Moody and Redick, and began compiling the known
research into a meta-analysis to quantify the gains and examine the
influence of various characteristics of the studies:
http://www.gwern.net/DNB%20meta-analysis

While it hasn't been easy, I have *generally* been able to get the
necessary data for each study from either the published papers or
when the papers omitted the exact
numbers I needed, from the authors. This was the case for some of the
early high-profile research - I was able to get data from the
researcher most identified with dual n-back, of course, Susanne Jaeggi.

But not for all the studies. A problem came up with
http://www.gwern.net/DNB%20FAQ#seidler-2010 - this was a very obscure
technical report issued mid-study about attempts to improve driving
ability in the elderly via brain games, funded by an agency at the
University of Michigan. It omits any numbers, and there have been no
followup publications I am aware of. This is probably because the
interim results were that there were no benefits to the brain games.
So I contacted the lead author, Seidler, for the IQ scores. She told
me she didn't have the data, but to contact one of her co-authors for
it - Jaeggi...

I did again, and this time Jaeggi was not happy to provide me data but
angrily turned me down, saying I should concern myself only with
published peer-reviewed papers. I don't know how familiar you are with
meta-analysis, but Jaeggi's suggestion would strike any meta-analyst
with horror because it is an invitation to publication bias, which
will be bad and biases all meta-analyses quite enough without actively
doing so, to the point where meta-analysis textbooks & guides will
urge you to seek out 'grey literature' like theses, unpublished
papers, technical reports, etc, specifically to counteract this
problem. (Ironically, the technical report in question was a null
result just like publication bias predicts.) I quoted one such text at
her, but it made no difference. She also cited her lab at University
of *Maryland*'s policy against sharing unpublished data; I pointed out
that that was not Michigan, and in any case, she was head of that lab
and could change that policy as she wished.

In the year or so since, neither Jaeggi nor Seidler has given me the
IQ test data, and my emails to the funding agency have gone
unanswered. My patience has run out, so I am considering more drastic
measures. Specifically, according to M-CASTL's policies for its
sponsored research http://m-castl.org/node/47

> All PIs are required to submit a final report for each research project by the end of the project period. Final reports should give a complete description of the problem, approach, methodology, findings, conclusions, and recommendations developed in the project. They should also completely document all data gathered, analyses performed, and results achieved.

So if the data wasn't in the paper, it should've been submitted with
the paper to M-CASTL. Or if they were lax and didn't get the data,
perhaps sheer embarrassment will compel Jaeggi to yield up the data.

The federal FOIA apparently doesn't apply directly to Michigan, but there is a
Michigan act directly inspired by the idea. I found the following
useful links on it:

- MI background: http://www.nfoic.org/michigan-foia-laws &
http://www.nfoic.org/michigan-foi-resources
- Full background for Michigan:
http://www.legislature.mi.gov/%28S%281qjevu55zgjkeomgfg2zjgjx%29%29/mileg.aspx?page=getobject&objectname=mcl-act-442-of-1976
- Sample request to modify: http://www.nfoic.org/michigan-sample-foia-request

My reasoning is that the paper was funded by M-CASTL, which is run by
the Transportation Research Institute, which is run by the University
of Michigan, which is run by the State of Michigan, which has a FOIA
analogue act, and does not require residency as far as I can tell. So
I should be able to file a request for the research data. Thoughts?

--
gwern

I'm out of my league here, both due to unfamiliarity with law and the deep novelty of this case. But here we go: does the State of Michigan have any significant stake in seeing Jaeggi succeed? University funding is valuable and runs on the coattails of the value of the program to the stakeholders. Would Michigan want to see an n-back research program succeed, to bring in outside interest?

On Sun, Jul 7, 2013 at 3:54 PM, FerrousFerriss
<kenneth.b...@gmail.com> wrote:
> I'm out of my league here, both due to unfamiliarity with law and the deep
> novelty of this case. But here we go: does the State of Michigan have any
> significant stake in seeing Jaeggi succeed? University funding is valuable
> and runs on the coattails of the value of the program to the stakeholders.
> Would Michigan want to see an n-back research program succeed, to bring in
> outside interest?

The answer is probably yes. A fair amount of the n-back research has
been done with or by the Jonides lab
http://www-personal.umich.edu/~jjonides/ which is active in non-n-back
training targeted at children and educational institutes; negative
information about n-back is, by extension, negative information about
the other WM research and its prospects.

--
gwern
http://www.gwern.net

Gwern with all the respect, I think you should show some respect here -

On Thu, Jul 11, 2013 at 2:40 PM, polar <pol...@gmail.com> wrote:
> Gwern with all the respect, I think you should show some respect here -

I disagree completely, and I am slightly sickened that you bring up
primate status games in the context of science. No 'respect' should be
shown. Everything should be assessed on its merits. Remember the motto
of the Royal Society.

If a researcher refuses to provide data without an excellent reason
like national security, they are not doing science. If they refuse to
provide data against their theory, they are not doing science, but
they are committing fraud and deceiving mankind.

> firstly, to gather the data is most tedious and expensive enterprise, and

It is not the researcher who bears the cost of deception; it is the
public who funded the research who bears the cost, it is the
publicly-funded schools shelling out thousands of dollars per student
for brain-training who bears the costs, it is the student given
worthless training who bears the cost, it is everyone everywhere who
learns of it and bases any belief on it.

I don't give a damn about the researcher. They are paid for what they
do. If they don't like providing their data and doing actual science,
they can go do something else! There are many replacements for them.

> nobody will give you this treasure for free. "Anybody" can do a
> meta-analyses,
> but very few people will apply and get the funding, because very few
> people have expert knowledge of the field and enough practical experiences.

There are multiple grant applications for every available grant. There
is no such limit. There is a vast oversupply of would-be researchers.

And if 'anybody' can do a meta-analysis, why hasn't it been done already?

> This is where discussion group enthusiast should recognize their limits,
> both of good taste, common sense and in legistation. Its questionable
> whether you can force access to anybody's data even after publication
> (journal should check if everything is all right, or other established
> reviewers - not you). And you definitely cant force anybodys data BEFORE
> research is published.

The study in question is not published, there were no indications that
it would be published, no one I have contacted has indicated it will
ever be published, and it has not been published in the 4 years or so
since it has been written up as a technical report. I should have
looked into FOIA at the time, before so much time had passed that the
data could be lost, but I didn't know then that I would be doing a
meta-analysis.

> And thanks to common sense, you have no chance
> to get them by appealing to freedom of information :)

Legally, it seems like it should work fine.

> Just relax, you did a nice job. Btw, have you published your methodology
> in detail?

Do you have any questions not answered on the existing page?

> Anyway, you present your opinions in your FAQ, enjoy that,
> and please dont make problems to people who have the same goal as you
> - widening our common knowledge about working memory training.

They are not widening 'common knowledge' if they are selectively
biasing released information. Systematic bias means that the more
'knowledge' one thinks one has, the more misleading & false one's
beliefs are.

--
gwern
http://www.gwern.net

I think this is a very good idea, and I think it has a decent chance of success if the data was indeed submitted (rather than just skipped over).  

On Fri, Jul 12, 2013 at 11:01 AM, D Kong <deliriou...@gmail.com> wrote:
> I think this is a very good idea, and I think it has a decent chance of
> success if the data was indeed submitted (rather than just skipped over).

I'm afraid it's very probable that the data was never submitted and
exists now, if it exists anywhere, solely on Jaeggi's hard drive.

> I'd be interested in helping out, but I don't have any particular knowledge;
> although I know at least one person which has filed FoIA requests before.
>
> Also to illustrate how powerful such requests can be...
> http://techcrunch.com/2013/04/30/enigma-makes-unearthing-and-sifting-through-public-data-a-breeze/

Interesting. The Seidler data wouldn't be in there, of course, but
they might have some other data of interest to me, like trace lithium
in drinking water data.

--
gwern
http://www.gwern.net

Yes, our perspectives apparently differ (and not only ours I imagine). While you are

That's very interesting link, thanks. Of course there's always problem with context and interpreting raw data, but still it looks great.

Dne pátek, 12. července 2013 17:01:45 UTC+2 DK napsal(a):

On Sat, Jul 13, 2013 at 6:57 AM, polar <pol...@gmail.com> wrote:
> Yes, our perspectives apparently differ (and not only ours I imagine). While
> you are
> talking about primate games, I wasnt (but could) talk about narcissism.
> You want to threat with jail people,

As far as I know, refusal to comply with a FOIA request is not a
criminal but a civil matter. And as I've already pointed out, they
could easily simply say the data was never submitted or has been lost,
so even if it were a criminal charge*, there could be no real threat
of jail here.

* it's really too bad that it isn't, the FOIA has done *amazing*
things for good governance and exposing corruption and crime in the
USA, even with government agencies scoffing at it

> who IN YOUR OPINION have some data

Not in my opinion. *They themselves wrote up the paper saying the
results were not statistically-significant*. Seriously, what the fuck,
Pontus? Do you have any idea what I'm talking about here? What, you
think I made up the existence of this data or this study? Hallucinated
or guessed it or something? 'My opinion'? Nothing to do with my
opinion, they told the world all about it.

> that COULD be in line with your (unscientific and one-sided btw) view, that
> brain training is "worthless".

Not could be, are. And my view is looking at this point to be far more
scientific than theirs.

It's funny that you claim my view that brain training is worthless
when it is the default assumption given the utter failure of all past
such research and literally the last study posted to this group was a
failure to find any meaningful transfer to grades or exams!

> You are right, that scientist are mostly paid by public (except private
> foundations,
> and except when they pay themselves).

Indeed.

> Btw usually its pretty small money and
> you always do a lot more work that you are paid for. Regarding your
> "oversupply
> wanabe scientist" - I'm really sorry, pardon me, but these always have at
> least
> the skill you have.

So? Did I ever say I was far more skilled than them? What does that
have to do with anything?

Shouldn't your reply have been something relevant to what I said, like
"actually, there are not hundreds of postdocs vying for every position
and desperate pressures to publish sexy research by means fair and
foul, despite the common beliefs and all evidence, and here's why..."?
Why the ad hominems?

> Btw I'm not putting energy into reviewing your
> metaanalyses,
> but I will not accept any of your conclusions, until they pass rigorous
> peer-reviewed
> process (both statistically and commonsensically) in some impacted journal,

'La la la la I can't hear you!'

I think it's very telling that - though I've been working on the
meta-analysis for more than a year now, have discussed it in detail,
and the results should be of *extreme* interest to anyone interested
in DNB - you have not read it.

> let alone because I know of your strong and long bias against N-back.

I did not start biased against N-back (feel free to go back and read
my early posts), reading the research and meta-analyzing the results
changed my views. "When the facts change, I change my mind. What do
you do, sir?"

> With what I dont agree, is the notion that anybody can get a grant. No,
> maybe
> one fifth or one tenth of applications will get it - as I said, only the
> very best
> scientist with serious relevant experiences and a excellent research plan.

Or the ones with the sexiest (==least likely to be true) results.

> And one more thing - I'm not some crazy constructivist relativist, but
> science really
> is done by people, and you can never take human aspect out of it -
> fortunately.
> Of course I am not advocating manipulating data or not publishing it.

Really? Because that seems to be exactly what you're defending here.

> I just want to say, that people (and scientist) do gestures of good will for
> each other
> every day, that are not enforceable by asertivity (nor law of course). And
> without
> respect (which includes not threatening one with jail for some small data
> that
> maybe exist, and maybe were not and wont be published) you will lose
> respect.

What is the point of 'respect' if you cannot get what you need?

Fortunately, I have no skin in the game and I do not fear "losing
respect" since Jaeggi or Nisbett or Buschkuehl will never be reviewing
a grant application or paper of mine or sitting on a committee judging
me.

> But maybe you crave every piece of your contemporary "public rights" and
> "open-source-dictature" for your metaanalyses so much,

And you resort to name calling again, ignoring the stakes. Here's a
question for you, Pontus: brain-training has been running for
something like a decade now. In that decade, how many millions in
revenue have been collectively consumed by Cogmed, Luminosity, Jungle
Memory, Brain Twister, etc? Remembering the critical letter we
discussed here in 2011 or 2010 or so which mentioned that these
companies were charging school districts hundreds or thousands of
dollars per student per year?

> that one shitty old
> piece
> of some data from maybe 30 people

It's 30 more than I had before.

> is worth threatening some "manipulative
> scientist".

Releasing data which favors your cause and not data which does not
favor your cause is the epitome of manipulativeness.

--
gwern
http://www.gwern.net

You are not talking to Pontus Gwern :), and I dont have time nor motivation to

On Sat, Jul 13, 2013 at 5:23 PM, polar <pol...@gmail.com> wrote:
> You are not talking to Pontus Gwern :)

Whups. My bad.

> Make yourself comfortable, call me quits for this
> moment,
> enjoy your uber-faq and meta-analyses, enjoy your clear and strong opinions,
> I'll enjoy my research, different versions of n-back created for this group,
> questionnaires,
> opinions and gains of many users,

Good old placebo & expectancy effects. 9.3 IQ points versus 2.5 -
they're a hell of a drug, to paraphrase Thompson.

> and definitely all the upcoming research
> in this exciting area of cognitive and mental improvements.

"Even if the community of inquiry is both too clueless to make any
contact with reality and too honest to nudge borderline findings into
significance, so long as they can keep coming up with new phenomena to
look for, the mechanism of the file-drawer problem alone will
guarantee a steady stream of new results. There is, so far as I know,
no _Journal of Evidence-Based Haruspicy_ filled, issue after issue,
with methodologically-faultless papers reporting the ability of
sheeps' livers to predict the winners of sumo championships, the
outcome of speed dates, or real estate trends in selected suburbs of
Chicago. But the difficulty can only be that the evidence-based
haruspices aren't trying hard enough, and some friendly rivalry with
the plastromancers is called for. It's true that none of these
findings will last forever, but this constant overturning of old ideas
by new discoveries is just part of what makes this such a dynamic time
in the field of haruspicy. Many scholars will even tell you that their
favorite part of being a haruspex is the frequency with which a new
sacrifice over-turns everything they thought they knew about reading
the future from a sheep's liver! We are very excited about the renewed
interest on the part of policy-makers in the recommendations of the
mantic arts... "

--Cosma Shalizi, "The Neutral Model of Inquiry (or, What Is the
Scientific Literature, Chopped Liver?)"
http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/698.html

> And after next
> five or ten years, one of us will, I believe gladly, give credit to an 2013
> hypothesis
> / opinions of the other. Until then, your principially limited metaanalyses
> (one
> version of nback,

You probably should have read it before criticizing it. I classify
studies into 3 kinds of n-back. (The estimates for non-DNB aren't very
sharp but that's because no one uses them.)

> few hours of training,

A very wide range of training times is represented, and there is no
dose-response.

> indicriminated personalities,

Would this be referring to the post-hoc personality analysis which
failed to replicate recently? And personalities should dilute the
effect, not make it disappear, and could not possibly explain the
dramatic effect of active/passive control groups.

> and
> probably
> other invariable factors) simply wont do the job for me or most other
> scientists
> regarding WM training in general, so you better cope with it.

No, for WM training in general, we merely have Melby-Lervåg & Hulme
2013's "Is Working Memory Training Effective? A Meta-Analytic Review"
(http://www.apa.org/pubs/journals/releases/dev-49-2-270.pdf) raining
on the parade.

--
gwern
http://www.gwern.net

Update: I finally heard back from a Professor Eby associated with
M-CASTL. It seems that they did not ever get a copy of the actual
research data. So, a FOIA-equivalent would be futile. That leaves
Jaeggi as the only person with the data from this null result,
apparently. Very frustrating.

--
gwern
http://www.gwern.net

On Fri, Sep 13, 2013 at 7:46 PM, Green <dmuc...@gmail.com> wrote:
>  So the question is: should Gwern want to include Jaeggi's data in a
> meta-analysis?

Yes; I want to include data from every study I know of which fits my
basic criteria of being randomized, using adaptive n-back, and
post-training IQ tests. To pick and choose is its own bias, and in
this case, the reason for the experiment being stopped halfway is
probably the lack of positive results, so I am far from worried about
this being some bogus experiment which turned in an inflated result -
if it had been doing that, why did they stop?

>   If I remember correctly, Jaeggi has done some shady things in her
> research. She's published a post-hoc analysis,

Which by definition doesn't affect the original data.

> and the original N-back study
> paper had used a timed version of the Ravens test. I thought there were some
> other things as well.

The timing stuff was suspicious, but it does not actually seem to have
been an issue according to my meta-analysis. Certainly the timing is
nowhere as important as active vs passive control groups turned out to
be.

>      If she's not a competent (or ethical) researcher, then the data itself
> could be suspect. There are a million ways her experiments could be
> improperly administered so that bad results come out that wouldn't show up
> in a spreadsheet full of raw data.

Certainly, but it's absurd to think we're dealing with a second Stapel
here. Jaeggi wasn't even the lead author on the Seidler experiment. I
don't think she is guilty of anything worse than some publication bias
or post hoc analysis, which are common sins in psychology.

--
gwern
http://www.gwern.net

On Fri, Sep 13, 2013 at 11:48 PM, Green <dmuc...@gmail.com> wrote:
> I once worked in a lab, years ago, and did hours of god-awful data analysis.
> To this day I have a hundred hours of fMRI records on a set of excel files
> on an old flashdrive. I often wonder why the PI never asked me to delete
> them - I guess she forgot. Anyway, if you can find someone who might have
> worked as a research assistant for her, they could have a copy of the data.

Highly unlikely - IQ tests aren't complex bulky extremely-expensive
data like fMRI scans are. The results would fit on a sheet of paper.
Even if a grad student handled the data collection, they wouldn't have
a copy by this point in time. I have a hard enough time contacting
people who just authored a paper!

Incidentally, does anyone know how to get a hold of Andrew Clouter,
who wrote "The Effects of Dual n-back Training on the Components of
Working Memory and Fluid Intelligence" which we discussed 2 weeks ago?
I can't find an email address for him, and my efforts to reach him on
Facebook and Google+ seem to have failed.

> You could write them and tell them that Jaeggi is undermining transparency,
> breaking the law

That wouldn't be true, though, as I already pointed out: Jaeggi is not
in any way breaking the law by declining to give me this data. There
are very limited sharing requirements for researchers in the US, and
few or none of them apply here. If MCASTL had a copy of the data, I
could conceivably force it out of them under the Michigan version of
FOIA, and if they ignored a valid FOIA request then there are
penalties/remedies prescribed by that law, but the whole point of this
exercise was establishing whether it was futile to bust out the FOIA
request or not.

--
gwern
http://www.gwern.net

 >Highly unlikely - IQ tests aren't complex bulky extremely-expensive
data like fMRI scans are. The results would fit on a sheet of paper.
Even if a grad student handled the data collection, they wouldn't have
a copy by this point in time. I have a hard enough time contacting
people who just authored a paper!

  Wrong. I was in the situation of the grad student collecting data in 2008, and I still have data from back then. I also have data from other smaller studies, which only consisted of single pages of data.  Also, I described a process for finding possible lab assistants during that period - look for people who got their names mentioned in publications as non-authors (we thank Joe Smith...blah blah)  Obviously, you have to make your own judgment calls about how to expend your time and energy and I think it was pretty obvious I wasn't pushing this as a really promising direction.
 
  
 

On Sun, Sep 15, 2013 at 10:03 PM, Green <dmuc...@gmail.com> wrote:
> I was in the situation of the grad student collecting data in 2008, and I
> still have data from back then. I also have data from other smaller studies,
> which only consisted of single pages of data.

Just because *you* kept the data doesn't mean anyone else does. Try
contact 4 or 5 of your fellow lab slaves from 2007 or 2008 and asking
for random sessions' data and see how well that goes - or whether you
can contact them at all.

--
gwern
http://www.gwern.net

I tend to agree with gwern on this - and digging around behind Jaeggi's back could have repercussions down the line if there's some kind of 'old boys' network in that field.

Obviously not. Humor is too easily misunderstood, so I never use it in
serious requests.

--
gwern
http://www.gwern.net

The IQ-police is out for blood.....