FWD: Questionable research practices surprisingly common

110 views
Skip to first unread message

unread,
May 28, 2012, 12:39:01 PM5/28/12
to Dual N-Back, Brain Training & Intelligence
As a reminder for some of the issues that crop up during discussion of
the research:

Abstract (from the article itself):
"Cases of clear scientific misconduct have received significant media
attention recently, but less flagrant transgressions of research norms
may be more prevalent and in the long run more damaging to the
academic enterprise. We surveyed over 2,000 psychologists about their
involvement in questionable research practices, using an anonymous
elicitation format supplemented by incentives for honest reporting.
The impact of incentives on admission rates was positive and greater
for practices that respondents judge to be less defensible. Using
three different estimation methods, we find that the proportion of
respondents that have engaged in these practices is surprisingly high
relative to respondents' own estimates of these proportions. Some
questionable practices may constitute the prevailing research norm."

From MedicalXpress:
"They found that a surprising number of people had engaged in
questionable research practices. For example, half the scientists
admitted to having only reported the experiments that gave the results
you wanted. ... Other questionable practices include deciding whether
to exclude data from a study after looking to see whether doing so
affects the results (43.4 percent of respondents) and reporting an
unexpected finding as if it had been expected all along (35 percent).
And 1.7 percent of scientists admitted to having faked their data."

"One possibility might be for psychological scientists to consider
instituting a system like that starting to be used in medical
research, in which journals will only accept articles for publication
if the study was registered before it began, with details about how it
would be executed. “I think psychologists are motivated to do good
science,” John says. “But these findings are disconcerting and signal
the need for reform.”"

http://medicalxpress.com/news/2012-05-surprisingly-common.html

If you want to read the article, the uncorrected MS (a pdf file) can
be found through a Google search of the full article title: "Measuring
the Prevalence of Questionable Research Practices with Incentives for
Truth-telling."

argumzio

jttoto2

unread,
May 28, 2012, 9:18:39 PM5/28/12
to Dual N-Back, Brain Training & Intelligence
Why am I not surprised? This study is discouraging but important
nonetheless. Good find, and certainly sends a message that we should
not accept everything that is thrown at us, even if it is published in
a respectable journal.

Aman Abdullahi Idle

unread,
May 28, 2012, 9:59:48 PM5/28/12
to brain-t...@googlegroups.com
So gwern and argumizo i have a question. U guys seem bright and u guys use a critical mind(others do too) and dont take things for granted. Does it pleasure u wen u discover errors in experiment design or methodology which casts doubt on iq gain from dual n back or other mental games? I ask because no one want a competitor, and u mite enjoy the position of being superior in iq than the rest and hope there's no nootropic that comes out that might make me reach ur level using doping.
Another thing im wondering is, do you ever think we'll find a novel way of increasing iq either through brain games or nootropics? I mean its 2012, we do space travel and can observe planets million of light years away but nothing widely accepted, proven, useful method of increasing iq.

Ps: im not taking a stance against dual n back, im just seein it from their perspective. I personally give it hope as my personal view is that good food, sleep, brain exercising might be useful.

Sent from my iPhone
> --
> You received this message because you are subscribed to the Google Groups "Dual N-Back, Brain Training & Intelligence" group.
> To post to this group, send email to brain-t...@googlegroups.com.
> To unsubscribe from this group, send email to brain-trainin...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/brain-training?hl=en.
>

unread,
May 28, 2012, 11:16:35 PM5/28/12
to Dual N-Back, Brain Training & Intelligence
On May 28, 8:59 pm, Aman Abdullahi Idle <aman.i...@googlemail.com>
wrote:
> Does it pleasure u wen u discover errors in experiment design or methodology which casts doubt on iq gain from dual n back or other mental games?

No, mine is generally of disbelief and surprise that I can find such
errors, mainly because I'm not a professional practicing in the field,
while these people, a number of them PhDs, are practitioners who
*should* know better.

> I ask because no one want a competitor, and u mite enjoy the position of being superior in iq than the rest and hope there's no nootropic that comes out that might make me reach ur level using doping.

Competitor? You're making a false assumption, Aman. If anything that
comes around that can do for I.Q. what you apparently believe I'm
afraid of happening, then I'd simply jump on board; in that scenario,
the order of things will remain much the same I.Q.-wise. That is,
increases in raw score (from brain training, drugs, or what have you)
do not correspond to increases in I.Q. (!) anyway. As a developed
human being, I've put behind me the ancient notions from my primate
ancestors that such a thing as demonstrating a fleeting and
inconsequential thing as "superiority" in the social domain is
remotely worth my time; however, I can't speak for others.

> Another thing im wondering is, do you ever think we'll find a novel way of increasing iq either through brain games or nootropics? I mean its 2012, we do space travel and can observe planets million of light years away but nothing widely accepted, proven, useful method of increasing iq.
>

Your question is stated as a false dilemma, for there could be any
number of possible ways to increase intelligence; e.g., there's
genetic engineering (eugenics), but that ain't PC stuff.... But to
answer directly to your question, using those two options, I don't
think there's much that can influence I.Q. directly; and if anything
did, for it really to alter I.Q., it would have to be selective and
not affect an entire population on which I.Q. is based, changing the
order of portions of the populace but not others, thereby changing
where the mean lies and what the variance is. In such a situation, the
mechanism of action should be *bleeding obvious* to boot.

> Ps: im not taking a stance against dual n back, im just seein it from their perspective. I personally give it hope as my personal view is that good food, sleep, brain exercising might be useful.
>

Just in case there's a hidden assumption here, I'll state this clearly
for the umpteenth time: I'm not against DNB. I'm against crappy
research into the subject, including research into brain training in
general, just like any sane, rational, competent person *should be*.

Your "personal view" isn't just that. We know very well that good
nutrition, diet, exercise, adequate sleep, a decent social life,
intellectual stimulation (reading, education, critical thinking,
etc.), and the like greatly increase the chances that one will be
living at a level that fully maximizes the expression and use of one's
in-born, latent potential. But it is wrong to say that something takes
place when it is so obviously far from being demonstrated as the case.

argumzio

whoisbambam

unread,
May 29, 2012, 1:30:08 AM5/29/12
to Dual N-Back, Brain Training & Intelligence

while i certainly wouldnt 'mind' increasing my iq, i dont think it is
possible.

i would like to improve my memory, however, and this *might* be
possible, esp. long-term memory or memory as it relates to academic
study, etc

and i would like to 'understand' more.......i can read something, and
i havent a clue. this is esp. true when it comes to reading research
articles and philosophy, etc

and i dont even know where to start (as i am interested in a gradual,
systematic, somewhat comprehensive approach/foundation)....





Gwern Branwen

unread,
May 29, 2012, 10:28:55 AM5/29/12
to brain-t...@googlegroups.com
On Mon, May 28, 2012 at 12:39 PM, ☉ <argu...@gmail.com> wrote:
> Cases of clear scientific misconduct have received significant media
> attention recently, but less flagrant transgressions of research norms
> may be more prevalent and in the long run more damaging to the
> academic enterprise. We surveyed over 2,000 psychologists about their
> involvement in questionable research practices, using an anonymous
> elicitation format supplemented by incentives for honest reporting.
> The impact of incentives on admission rates was positive and greater
> for practices that respondents judge to be less defensible. Using
> three different estimation methods, we find that the proportion of
> respondents that have engaged in these practices is surprisingly high
> relative to respondents' own estimates of these proportions. Some
> questionable practices may constitute the prevailing research norm

A good addition to http://www.gwern.net/DNB%20FAQ#fn51

On Mon, May 28, 2012 at 9:59 PM, Aman Abdullahi Idle
<aman...@googlemail.com> wrote:
> So gwern and argumizo i have a question. U guys seem bright and u guys use a critical mind(others do too) and dont take things for granted. Does it pleasure u wen u discover errors in experiment design or methodology which casts doubt on iq gain from dual n back or other mental games?

Do proponents of DNB like Pontus or polar take pleasure in knowing the
truth about how IQ is mutable and people like me are biased in favor
of fixed mindsets, and also in knowing the truth about how binaural
beats/shrimp coloring/coconut oil/etc. will transform your life? If
you are not worried about them, why worry about me?

--
gwern
http://www.gwern.net

polar

unread,
May 30, 2012, 6:52:08 AM5/30/12
to brain-t...@googlegroups.com
I dont understand how you meant this, but I think its perfectly ok to have
fans of dnb and fans of null hypothesis. Its despising one or another what
bothers me, and pretending that definitive answers are already here. And
for the record - I'm not interested in researching binaural beats or shrimp
oil (or nootropics for that matter, except maybe omega3).

Attila

unread,
May 30, 2012, 7:20:40 AM5/30/12
to Dual N-Back, Brain Training & Intelligence
Aside from the fact that IQ is a total BS measuring tool, why does
everyone seem to think it's impossible to become more intelligent? I
for one admittedly used to be a dumbass. Well... at least compared to
where I am now. No wait... a dumbass.

IQ is based in large on your knowledge base which everyone knows you
can increase. Just get a subscription to mental floss, toss in the
discovery and history channels, listen to some TEDtalks and bam,
instant score of 130+. I found it really surprising so little of the
IQ test used fluid intelligence which would be MY basis for
intelligence. Any boob can recite facts, I'm living proof.

As for fluid intelligence I know FOR A FACT it can be increased. There
is (IMHO) a major correlation between gf and creativity. The ability
to look at an object/situation in terms of potential rather than the
standard acceptance. In essence the ability to think outside the box.
While it takes some time this is very much trainable, I taught myself
by accident and then taught my best friend intentionally because he
was so impressed. In his words-

"When I think of a problem, I think of the logical steps from A to B
to C. When you think of a problem you jump to the furthest point
possible and wind through what seems like random ideas, somehow
landing on pure genius."

Don't take things for granted. If you limit yourself to the basic idea
that everything has a limited amount of uses or rules then the world
is like walking down a hall, one way to go. With no rules you can run
wherever the hell you want. A world of pure potential is a world
without limitations.



Quick note to Gwern, have you ever considered designing the perfect
study?



On May 29, 8:28 am, Gwern Branwen <gwe...@gmail.com> wrote:
> On Mon, May 28, 2012 at 12:39 PM, ☉ <argum...@gmail.com> wrote:
> > Cases of clear scientific misconduct have received significant media
> > attention recently, but less flagrant transgressions of research norms
> > may be more prevalent and in the long run more damaging to the
> > academic enterprise. We surveyed over 2,000 psychologists about their
> > involvement in questionable research practices, using an anonymous
> > elicitation format supplemented by incentives for honest reporting.
> > The impact of incentives on admission rates was positive and greater
> > for practices that respondents judge to be less defensible. Using
> > three different estimation methods, we find that the proportion of
> > respondents that have engaged in these practices is surprisingly high
> > relative to respondents' own estimates of these proportions. Some
> > questionable practices may constitute the prevailing research norm
>
> A good addition tohttp://www.gwern.net/DNB%20FAQ#fn51
>
> On Mon, May 28, 2012 at 9:59 PM, Aman Abdullahi Idle
>

Aman Idle

unread,
May 30, 2012, 7:47:40 AM5/30/12
to brain-t...@googlegroups.com
can you teach me the way you learned by "accident" and taught your friend attila?

Gwern Branwen

unread,
May 30, 2012, 9:32:45 AM5/30/12
to brain-t...@googlegroups.com
On Wed, May 30, 2012 at 7:20 AM, Attila <brains...@gmail.com> wrote:
> There
> is (IMHO) a major correlation between gf and creativity.

Really? I was under the impression that there wasn't, to the extent
that psychologists have developed reasonable measures of creativity. I
believe we've discussed this before in the context of latent
inhibition & WM, and the personality factor Openness.

> Quick note to Gwern, have you ever considered designing the perfect
> study?

No. It would never be implemented, and my time is better spent doing
things like trying to understand existing studies.

--
gwern
http://www.gwern.net

unread,
May 30, 2012, 5:33:53 PM5/30/12
to Dual N-Back, Brain Training & Intelligence
On May 30, 6:20 am, Attila <brainstobr...@gmail.com> wrote:
> As for fluid intelligence I know FOR A FACT it can be increased.

O RLY?

argumzio

unread,
May 30, 2012, 5:42:11 PM5/30/12
to Dual N-Back, Brain Training & Intelligence
On May 30, 8:32 am, Gwern Branwen <gwe...@gmail.com> wrote:
> Really? I was under the impression that there wasn't, to the extent
> that psychologists have developed reasonable measures of creativity.

Actually, based on any serious meaning of the term "creativity" as
understood operationally, in the lab, it is more or less just a skill
or set of skills combined with adequate fluid ability. This should hit
the fact home that g (the general factor) expresses itself in just
about everything one does. However, for the serious "creativity" of
the geniuses we likely have in mind, it most certainly doesn't, since
a number of other factors thereby become involved, such as personality
variables in tandem with I.Q.

For the sake of a better term, "laboratory creativity" (the first
kind) is really a test of aptitude(s). Think back to examples like
finding x number of ways to use a brick or something. In that
scenario, you're dealing more with fluency and verbal ability. But
would a truly "creative" person be concerned with enumerating x number
of ways to use a brick? In other words, this kind of "laboratory
creativity" fails on the level of ecological validity every time. But
that isn't to say it can't be done properly....

Similarly with that bit about the nine dots and solving it with four
lines: that is not a valid measure of creativity, and most certainly a
measure of fluid (visuospatial) ability. E.g., there are even a number
of one-line solutions, one of which immediately came to me, and I had
no need of electrical stimulation to my brain to realize it.

argumzio

Attila

unread,
May 30, 2012, 6:03:07 PM5/30/12
to Dual N-Back, Brain Training & Intelligence
ok. Maybe my definition of creativity is wrong.

FLUID INTELLIGENCE

Web definitions google search

The capacity to think logically and solve problems in novel
situations, independent of acquired knowledge
en.wiktionary.org/wiki/fluid_intelligence

an abstract form of intelligence that includes the ability to analyse
complex relationships, reason and find solutions to problems.
www.tuition.com.hk/psychology/f.htm

The part of intelligence which involves the use, as opposed to the
acquisition, of information.
allpsych.com/dictionary/dictionary2.html

is the basic power of reasoning and problem solving. It produces
deduction, induction, reasoning, and understanding of the
relationships between different ideas. ...
college.cengage.com/psychology/bernstein/psychology/6e/stude

-----

Now as for creativity I do not mean your ability to draw or write a
poem. That's artistic skill. Creativity is the means to come up with a
solution that is not obvious. to think outside the box. To piece
things together in a new way that you have never seen before. I am
talking about creative problem solving.

Classic example most of you have seen: you are shown a table pushed
against a wall. On this table are a candle, a box of tacks and a box
of matches. Your task is to attach the candle to the wall in a way
that it will not drip wax on the table.
If you think about it long enough most everyone will come up with the
answer. Dump the tacks out of the box, use the tacks to adhere the box
to the wall and put the candle in the box.

Abstract thinking, or a creative solution. Could this not also be
described as-

'The capacity to think logically and solve problems in novel
situations, independent of acquired knowledge'

or any of the other quoted definitions? I am not claiming to be
correct, simply stating an argument. I am the newbie on this board but
the definitions seem to match. I would rather you correct me should I
be wrong than stick stubbornly to my ignorance :)

And Gwern I only asked about your creating a study because usually
those who see the mistakes are the most likely to find the solution. I
have nothing but respect for you, my apologies if my writing came off
as otherwise

Attila

unread,
May 30, 2012, 6:05:19 PM5/30/12
to Dual N-Back, Brain Training & Intelligence
> can you teach me the way you learned by "accident" and taught your friend
> attila?


I'm actually writing a paper on creativity, as soon as it is done I
will post it. Actually I am now hoping this argument may shed new
light on my view of the subject :)

Attila

unread,
May 30, 2012, 6:43:14 PM5/30/12
to Dual N-Back, Brain Training & Intelligence
oops, sorry gwern looks like you posted same time as I was replying,
sorry I missed it.

I followed your post other than this

> However, for the serious "creativity" of
> the geniuses we likely have in mind, it most certainly doesn't, since
> a number of other factors thereby become involved, such as personality
> variables in tandem with I.Q.

I'm not sure I follow. If I understand right you are talking about
people like Einstein and Tesla? Genius level intellects with highly
creative inventions/discoveries, right? What is different between
their creativity and regular lab creativity?

Sure in the dot test there is a visuospatial solution. And like you
said many solutions (fold the paper, infinite line, ect...) but really
I would consider this a very dry form of the same reasoning.

Whether I'm solving the dot test or theorizing about a way to increase
cellular respiration I analyze the given variables, look at what can
effect those variables and then start plugging in ideas until
something fits. With the dot test I have far fewer variables to
consider, all of them visual, but the process is in essence the same.
Reply all
Reply to author
Forward
0 new messages