Publish and Be wrong
Florence Baingana
Publish and be wrong
Oct 9th 2008
From The Economist print edition
One group of researchers thinks headline-grabbing scientific reports are
the most likely to turn out to be wrong.
IN ECONOMIC theory the winner's curse refers to the idea that someone
who places the winning bid in an auction may have paid too much.
Consider, for example, bids to develop an oil field. Most of the offers
are likely to cluster around the true value of the resource, so the
highest bidder probably paid too much.
The same thing may be happening in scientific publishing, according to a
new analysis. With so many scientific papers chasing so few pages in the
most prestigious journals, the winners could be the ones most likely to
oversell themselves-to trumpet dramatic or important results that later
turn out to be false. This would produce a distorted picture of
scientific knowledge, with less dramatic (but more accurate) results
either relegated to obscure journals or left unpublished.
In Public Library of Science (PloS) Medicine, an online journal, John
Ioannidis, an epidemiologist at Ioannina School of Medicine, Greece, and
his colleagues, suggest that a variety of economic conditions, such as
oligopolies, artificial scarcities and the winner's curse, may have
analogies in scientific publishing.
Dr Ioannidis made a splash three years ago by arguing, quite
convincingly, that most published scientific research is wrong. Now,
along with Neal Young of the National Institutes of Health in Maryland
and Omar Al-Ubaydli, an economist at George Mason University in Fairfax,
Virginia, he suggests why.
It starts with the nuts and bolts of scientific publishing. Hundreds of
thousands of scientific researchers are hired, promoted and funded
according not only to how much work they produce, but also to where it
gets published. For many, the ultimate accolade is to appear in a
journal like Nature or Science. Such publications boast that they are
very selective, turning down the vast majority of papers that are
submitted to them.
Picking winners
The assumption is that, as a result, such journals publish only the best
scientific work. But Dr Ioannidis and his colleagues argue that the
reputations of the journals are pumped up by an artificial scarcity of
the kind that keeps diamonds expensive. And such a scarcity, they
suggest, can make it more likely that the leading journals will publish
dramatic, but what may ultimately turn out to be incorrect, research.
Dr Ioannidis based his earlier argument about incorrect research partly
on a study of 49 papers in leading journals that had been cited by more
than 1,000 other scientists. They were, in other words, well-regarded
research. But he found that, within only a few years, almost a third of
the papers had been refuted by other studies. For the idea of the
winner's curse to hold, papers published in less-well-known journals
should be more reliable; but that has not yet been established.
The group's more general argument is that scientific research is so
difficult-the sample sizes must be big and the analysis rigorous-that
most research may end up being wrong. And the "hotter" the field, the
greater the competition is and the more likely it is that published
research in top journals could be wrong.
There also seems to be a bias towards publishing positive results. For
instance, a study earlier this year found that among the studies
submitted to America's Food and Drug Administration about the
effectiveness of antidepressants, almost all of those with positive
results were published, whereas very few of those with negative results
were. But negative results are potentially just as informative as
positive results, if not as exciting.
The researchers are not suggesting fraud, just that the way scientific
publishing works makes it more likely that incorrect findings end up in
print. They suggest that, as the marginal cost of publishing a lot more
material is minimal on the internet, all research that meets a certain
quality threshold should be published online. Preference might even be
given to studies that show negative results or those with the highest
quality of study methods and interpretation, regardless of the results.
It seems likely that the danger of a winner's curse does exist in
scientific publishing. Yet it may also be that editors and referees are
aware of this risk, and succeed in counteracting it. Even if they do
not, with a world awash in new science the prestigious journals provide
an informed filter. The question for Dr Ioannidis is that now his latest
work has been accepted by a journal, is that reason to doubt it?
------------------------------------------------------------------------------------------------------------------------------------
MUIPH Mail Disclaimer: 10/10/2008
The email and any files transmitted with it are confidential and intended solely for the recipient. If you are not the named addressee you should not disseminate, distribute, copy or alter this email.
Any views or opinions presented in this email are solely those of the author and do not represent those of THE MAKERERE UNIVERSITY SCHOOL OF PUBLIC HEALTH (MUSPH).
Boris Budosan
Well, what I would say is that so many nice reports and articles get published, but as a person who has spent quite a some time as a frontline worker in the field, I have seen a huge gap between realities in the field and published reports, studies, articles and policies.
It is defintivelly necessary to publish and establish the evidence base on paper, but it seems that some paper work is far away from the reality on the ground.
I have an impression that some people do field work and implement mostly without publishing, and some people are focused exclusivelly on publishing.
I think more balance between field and office work is needed; frontline workers should start publishing and experts should start spending more time in the field,
|
Florence Baingana
From: Boris Budosan [mailto:bbud...@yahoo.com]
Sent: Fri 10/10/2008 9:29 PM
To: Academic Staff; mentalhe...@googlegroups.com; Florence Baingana
Subject: Re: [MHFS] Publish and Be wrong
Disclaimer: 11/10/2008
The email and any files transmitted with it are confidential and intended solely for the recipient, recipient / recipient. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and do not represent those of THE MAKERERE UNIVERSITY SCHOOL OF PUBLIC HEALTH (MUSPH).
Boris Budosan
I think that field workers should start with the proper assessment of the situation in the field.
There is still no consenus on what would be the best instruments to do that. What is usually done is some kind of combined qualitative & quantiattive assessment. Still, many good qustionnaires used in the West have not been either pilot-tested or adapted for developing countries. Proper monitoring / evaluation of programs is essential. Ideally, control group should be involved, not just pre-post intervention evaluation.
I think field workers who spend at least couple of months or more in the field are best positioned to supply evidence-base on mental health / psychosocial interventions and their success or failure.
Experts usually do not spend more than couple of weeks up to one month in the field, and I am not sure this time is enough to get a grasp on how mental health system in one country is really functioning. Please, correct me if I am wrong. |
SHANTHI THAMBAIAH
| Dear Baingana , Budosan and other friends. Feel really happy to be part of this forum. Usually I don't have full faith in forums, where not much realities come to the stage. But surprisingly lot of importance is given so far for the realities and Happy to see lot of people in the sector are concern over it. Budosan I couldn't resist myself to say what I think is very much similar to your thoughts about the recherches, connect the realities to the Publishings, and bring the practitioners or the front line workers in more role taking etc. Apart from this thoughts I would like to share more concerns on researches in general, it can be quantitative or qualitative or participatory (which is very famous in the sector now) or what ever, there are some ethics in practice without meanings. 1. Getting a consent : How far it has a meaning? usually when outside researchers come to a country or even the local research bodies they use an organization whom they know or through a funding agency use an organization which the Funders partner organization in implementation or in funding line . When a Funding agency go to a local organization and ask for a group of people to do research , does that agency have any power to say no? And when they go to the community they work with and ask for a consent to be part of an research does the community have any power to say no?? Is it possible for a community in a fragile state or under 30 years of war for example, to stand up and say no to a research where they know for sure this mite lead to some 'stops' of assistance to them in the future. I seriously don't have an answer in what way we can get a consent , but I feel when something has no meaning or value Why we have to do it and get a 'yes' ,(where we know for sure there won't be any 'no') because the research ethics talks about consent 2. The very nagging issue in my head is to tell the communities we use for research ' we will come back to you before we finalize the research and you will definitely get access to the final report or the publish'!!: this is not only promised thousand times to communities by the foreign missions but also by the local missions. But how many communities get the opportunities to see the researches again before they finalize the report or seen what colour the final report is?? I still couldn't find the answer why we tell this to the communities? Is it because the research ethics talks about this or do we really mean it?? I strongly feel the community should be consult again and again and again before we come to the final reporting and go to the wider world to read and get a view about certain community. Its very wrong to give a perception about a community to the world where that community doesn't feel the same or have an ownership over it. Research is important , no doubt about that, but how we are conducting a research is far more important than the validation of the data or the final report. For whom the research?? is it for us?? or it is for the people who are suffering and expecting better solutions, services etc? and who's data we are using?? is it ours ? or the community owns it and we are just a guardians for the time being? I may be wrong in my views, but I think we all have the responsibility when we use someones time, knowledge , experience , information etc to acknowledge that. But acknowledgment should not end on an A4 glossy sheet in times new roman size 12 saying we thank this community for the ...... At least it should go beyond to an extension where we send the pictures we took the children playing happily or the mother feeding a child in a community meeting , but still trying to participate and contribute to our research. How many of us sent copies of pictures we took the children playing, using fancy digital cameras where the children not even seen a normal camera and hardly have pictures of themselves, giving pose for us with smiles in their face when shells and bombers took over the job of thunder? How many of us took the permission of a woman in her dressing gown with no proper buttons , holding the top part with hands with embarrassment before her picture go for the final report or the publish? Isn't it an ethics of a research not to hurt someones dignity and not damage a community image?? These are the questions pressing me hard every day in my work. I am very much open to hear other voices on this. Thanks Shanthi Shanthi  Sivaprashanthi. Thambaiah J 2/8 Andersan Flats Colombo- 05
--- On Sat, 11/10/08, Boris Budosan <bbud...@yahoo.com> wrote: |
Rachel Jenkins
Likewise, I think this is a very good forum. I agree about the previous issues raised. Another concern I have are the research studies from rich countries which evaluate an intervention which is not affordable or practicable in low income countries. I suppose such studies of ARVs are now implemented with large amounts of donor money, but it concerns me in mental health as I feel we are so unlikely to get large amounts of money that it would be more ethical to focus the research on practical solutions.
Rachel
Rachel Jenkins
----- Original Message -----
From: "Florence Baingana" <fbai...@musph.ac.ug>
To: bbud...@yahoo.com, "Academic Staff" <Academ...@musph.ac.ug>, mentalhe...@googlegroups.com
Sent: Saturday, 11 October, 2008 10:33:35 AM GMT +00:00 GMT Britain, Ireland, Portugal
Subject: [MHFS] Re: Publish and Be wrong
rebecca...@gmail.com
fieldworkers to publish their research. I spent just over three years
working for the Jesuit Refugee Service in Kakuma refugee camp,
northern Kenya, and also saw a great deal of work being done and not
written up or shared. Fieldworkers tend to be too busy providing
services to write about their work, and it can be difficult to write
for publication without access to resources, like libraries and
internet. I submitted a paper for publication while I was in Kakuma
and it ws rejected because I hadn't reviewed the literature thoroughly
enough.
When I left, I applied for 9 months funding from the Economic and
Social Research Council to write up three projects I worked on when I
was with JRS, and I got it - which suggests that at least some funding
agencies recognise the need for fieldworkers' experiences to be
shared, and are willing to fund this process. I recognise that it is
easier for some people than others to apply for research funding, but
it might be worth trying this option. The other possibility, as
Rachel suggests, is for fieldworkers to link with an academic
institution, which can provide access to resources and other forms of
support.
Rebecca
On Oct 12, 2:10 pm, Rachel Jenkins <rac...@olan.org> wrote:
> In Zanzibar it is an essential part of basic nurse training to do a research project, and many are good- we need to encourage it to continue as part of their working roles. I think research has become too high tech and health workers are put off or discouraged. We need more journals that will publish low tech research from local health workers. and perhaps research advisors to help with sampling and study design. Rachel
>
>
>
> ----- Original Message -----
> From: "Florence Baingana" <fbaing...@musph.ac.ug>
> To: bbudo...@yahoo.com, "Academic Staff" <AcademicSt...@musph.ac.ug>, mentalhe...@googlegroups.com
> Sent: Saturday, 11 October, 2008 10:33:35 AM GMT +00:00 GMT Britain, Ireland, Portugal
> Subject: [MHFS] Re: Publish and Be wrong
>
> I like, and agree with those comments. How do we empower those who spend a lot of time int he filed to do more research, I believe that way, we do get the people in the science labs and class rooms to actually do more work int he field.
>
> Florence
>
> Sent: Fri 10/10/2008 9:29 PM
> To: Academic Staff; mentalhe...@googlegroups.com; Florence Baingana
> Subject: Re: [MHFS] Publish and Be wrong
>
> Well, what I would say is that so many nice reports and articles get published, but as a person who has spent quite a some time as a frontline worker in the field, I have seen a huge gap between realities in the field and published reports, studies, articles and policies.
>
> It is defintivelly necessary to publish and establish the evidence base on paper, but it seems that some paper work is far away from the reality on the ground.
> I have an impression that some people do field work and implement mostly without publishing, and some people are focused exclusivelly on publishing.
>
> I think more balance between field and office work is needed; frontline workers should start publishing and experts should start spending more time in the field,
>
> Boris Budosan
>
>
> It seems likely that the danger of a winner's curse does exist in
> scientific publishing. Yet it may also be that editors and referees are
> aware of this risk, and succeed in counteracting it. Even if they do
> not, with a world awash in new science the prestigious journals provide
> an informed filter. The question for Dr Ioannidis is that now his latest
> work has been accepted by a journal, is that reason to doubt it?
>
> MUIPH Mail Disclaimer: 10/10/2008
>
> The email and any files transmitted with it are confidential and intended
> solely for the recipient. If you are not the named addressee you should not
> disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the author
> and do not represent those of THE MAKERERE UNIVERSITY SCHOOL OF PUBLIC HEALTH
> (MUSPH).
>
>
> The email and any files transmitted with it are confidential and intended solely for the recipient, recipient / recipient. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and do not represent those of THE MAKERERE UNIVERSITY SCHOOL OF PUBLIC HEALTH (MUSPH).- Hide quoted text -
>
> - Show quoted text -