A few months ago, we discussed the replicability issues associated with high-impact publications in stem cell research on this blog. Some of the most exciting and most widely cited papers using adult stem cells could not be replicated during subsequent studies, and resulted in a lot of conflicting data and frustration among stem cell scientists. However, this is not just an affliction of stem cell research, but it also applies to many other areas of research, such as cancer biology.
Here is an excerpt from an article I wrote for Salon.com about the issues of reproducibility in pre-clinical cancer research:
The cancer researchers Glenn Begley and Lee Ellis made a rather remarkable claim last year. In a commentary that analyzed the dearth of efficacious novel cancer therapies, they revealed that scientists at the biotechnology company Amgen were unable to replicate the vast majority of published pre-clinical research studies. Only 6 out of 53 landmark cancer studies could be replicated, a dismal success rate of 11%! The Amgen researchers had deliberately chosen highly innovative cancer research papers, hoping that these would form the scientific basis for future cancer therapies that they could develop. It should not come as a surprise that progress in developing new cancer treatments is so sluggish. New clinical treatments are often based on innovative scientific concepts derived from pre-clinical laboratory research. However, if the pre-clinical scientific experiments cannot be replicated, it would be folly to expect that clinical treatments based on these questionable scientific concepts would succeed.
Reproducibility of research findings is the cornerstone of science. Peer-reviewed scientific journals generally require that scientists conduct multiple repeat experiments and report the variability of their findings before publishing them. However, it is not uncommon for researchers to successfully repeat experiments and publish a paper, only to learn that colleagues at other institutions can’t replicate the findings. This does not necessarily indicate foul play. The reasons for the lack of reproducibility include intentional fraud and misconduct, yes, but more often it’s negligence, inadvertent errors, imperfectly designed experiments and the subliminal biases of the researchers or other uncontrollable variables.
Clinical studies, of new drugs, for example, are often plagued by the biological variability found in study participants. A group of patients in a trial may exhibit different responses to a new medication compared to patients enrolled in similar trials at different locations. In addition to genetic differences between patient populations, factors like differences in socioeconomic status, diet, access to healthcare, criteria used by referring physicians, standards of data analysis by researchers or the subjective nature of certain clinical outcomes – as well as many other uncharted variables – might all contribute to different results.
You can read the complete article here at Salon.com. It is important to note that the the replicability issues were identified in pre-clinical research, i.e. lab bench studies and were not the result of varying responses between patients that often plague clinical research. Pre-clinical cancer researchers use molecular and cellular techniques that are commonly used in neuroscience, immunology, stem cell biology and many other areas of the life sciences. Therefore, all of these areas of biological research may have similarly poor rates of replicability.
This highlights the importance of conducting studies that attempt to replicate previously published data. The recent award of $1.3 million to the Reproducibility Initiative by the Laura and John Arnold Foundation is small, yet important step in the right direction, because funding for replicability testing is very hard to obtain.