'Robot scientist' Eve finds that less than one-third of scientific results are reproducible
4 views
Skip to first unread message
rael-science
Apr 21, 2022, 1:41:10 PMApr 21
to rael-science@googlegroups.com (rael-science)
'Robot scientist' Eve finds that less than one-third of scientific results are reproducible
Researchers
have used a combination of automated text analysis and the "robot
scientist" Eve to semi-automate the process of reproducing research
results. The problem of lack of reproducibility is one of the biggest
crises facing modern science.
The
researchers, led by the University of Cambridge, analyzed more than
12,000 research papers on breast cancer cell biology. After narrowing
the set down to 74 papers of high scientific interest, less than one
third—22 papers—were found to be reproducible. In two cases, Eve was
able to make serendipitous discoveries.
The
results, reported in the Journal of the Royal Society Interface,
demonstrate that it is possible to use robotics and artificial
intelligence to help address the reproducibility crisis.
A
successful experiment is one where another scientist, in a different
laboratory under similar conditions, can achieve the same result. But
more than 70% of researchers have tried and failed to reproduce another
scientist's experiments, and more than half have failed to reproduce
some of their own experiments: This is the reproducibility crisis.
"Good
science relies on results being reproducible; otherwise, the results
are essentially meaningless," said Professor Ross King from Cambridge's
Department of Chemical Engineering and Biotechnology, who led the
research. "This is particularly critical in biomedicine: If I'm a
patient and I read about a promising new potential treatment, but the
results aren't reproducible, how am I supposed to know what to believe?
The result could be people losing trust in science."
Several
years ago, King developed the robot scientist Eve, a computer/robotic
system that uses techniques from artificial intelligence (AI) to carry
out scientific experiments.
"One
of the big advantages of using machines to do science is they're more
precise and record details more exactly than a human can," said King.
"This makes them well-suited to the job of attempting to reproduce
scientific results."
As
part of a project funded by DARPA, King and his colleagues from the UK,
US and Sweden designed an experiment that uses a combination of AI and
robotics to help address the reproducibility crisis, by getting
computers to read scientific papers and understand them, and getting Eve
to attempt to reproduce the experiments.
For
the current paper, the team focused on cancer research. "The cancer
literature is enormous, but no one ever does the same thing twice,
making reproducibility a huge issue," said King. "Given the vast sums of
money spent on cancer research, and the sheer number of people affected
by cancer worldwide, it's an area where we urgently need to improve
reproducibility."
From
an initial set of more than 12,000 published scientific papers, the
researchers used automated text mining techniques to extract statements
related to a change in gene expression in response to drug treatment in
breast cancer. From this set, 74 papers were selected.
Two
different human teams used Eve and two breast cancer cell lines and
attempted to reproduce the 74 results. Statistically significant
evidence for repeatability was found for 43 papers, meaning that the
results were replicable under identical conditions; and significant
evidence for reproducibility or robustness was found in 22 papers,
meaning the results were replicable by different scientists under
similar conditions. In two cases, the automation made serendipitous
discoveries.
While
only 22 out of 74 papers were found to be reproducible in this
experiment, the researchers say that this does not mean that the
remaining papers are not scientifically reproducible or robust. "There
are lots of reasons why a particular result may not be reproducible in
another lab," said King. "Cell lines can sometimes change their behavior
in different labs under different conditions, for instance. The most
important difference we found was that it matters who does the
experiment, because every person is different."
King
says that this work shows that automated and semi-automated techniques
could be an important tool to help address the reproducibility crisis,
and that reproducibility should become a standard part of the scientific
process.
"It's
quite shocking how big of an issue reproducibility is in science, and
it's going to need a complete overhaul in the way that a lot of science
is done," said King. "We think that machines have a key role to play in
helping to fix it."
Explore further
More information: Testing the Reproducibility and Robustness of the Cancer Biology Literature by Robot, Journal of the Royal Society Interface (2022). DOI: 10.1098/rsif.2021.0821. rsif.royalsocietypublishing.or … .1098/rsif.2021.0821
Reply all
Reply to author
Forward
0 new messages