To Err Is Human, To Study Errors Is Science

The family of cholesterol lowering drugs known as ‘statins’ are among the most widely prescribed medications for patients with cardiovascular disease. Large-scale clinical studies have repeatedly shown that statins can significantly lower cholesterol levels and the risk of future heart attacks, especially in patients who have already been diagnosed with cardiovascular disease. A more contentious issue is the use of statins in individuals who have no history of heart attacks, strokes or blockages in their blood vessels. Instead of waiting for the first major manifestation of cardiovascular disease, should one start statin therapy early on to prevent cardiovascular disease?

If statins were free of charge and had no side effects whatsoever, the answer would be rather straightforward: Go ahead and use them as soon as possible. However, like all medications, statins come at a price. There is the financial cost to the patient or their insurance to pay for the medications, and there is a health cost to the patients who experience potential side effects. The Guideline Panel of the American College of Cardiology (ACC) and the American Heart Association (AHA) therefore recently recommended that the preventive use of statins in individuals without known cardiovascular disease should be based on personalized risk calculations. If the risk of developing disease within the next 10 years is greater than 7.5%, then the benefits of statin therapy outweigh its risks and the treatment should be initiated. The panel also indicated that if the 10-year risk of cardiovascular disease is greater than 5%, then physicians should consider prescribing statins, but should bear in mind that the scientific evidence for this recommendation was not as strong as that for higher-risk individuals.

 

Oops button – via Shutterstock

Using statins in low risk patients

The recommendation that individuals with comparatively low risk of developing future cardiovascular disease (10-year risk lower than 10%) would benefit from statins was met skepticism by some medical experts. In October 2013, the British Medical Journal (BMJ) published a paper by John Abramson, a lecturer at Harvard Medical School, and his colleagues which re-evaluated the data from a prior study on statin benefits in patients with less than 10% cardiovascular disease risk over 10 years. Abramson and colleagues concluded that the statin benefits were over-stated and that statin therapy should not be expanded to include this group of individuals. To further bolster their case, Abramson and colleagues also cited a 2013 study by Huabing Zhang and colleagues in the Annals of Internal Medicine which (according to Abramson et al.) had reported that 18 % of patients discontinued statins due to side effects. Abramson even highlighted the finding from the Zhang study by including it as one of four bullet points summarizing the key take-home messages of his article.

The problem with this characterization of the Zhang study is that it ignored all the caveats that Zhang and colleagues had mentioned when discussing their findings. The Zhang study was based on the retrospective review of patient charts and did not establish a true cause-and-effect relationship between the discontinuation of the statins and actual side effects of statins. Patients may stop taking medications for many reasons, but this does not necessarily mean that it is due to side effects from the medication. According to the Zhang paper, 17.4% of patients in their observational retrospective study had reported a “statin related incident” and of those only 59% had stopped the medication. The fraction of patients discontinuing statins due to suspected side effects was at most 9-10% instead of the 18% cited by Abramson. But as Zhang pointed out, their study did not include a placebo control group. Trials with placebo groups document similar rates of “side effects” in patients taking statins and those taking placebos, suggesting that only a small minority of perceived side effects are truly caused by the chemical compounds in statin drugs.

 

Admitting errors is only the first step

Whether 18%, 9% or a far smaller proportion of patients experience significant medication side effects is no small matter because the analysis could affect millions of patients currently being treated with statins. A gross overestimation of statin side effects could prompt physicians to prematurely discontinue medications that have been shown to significantly reduce the risk of heart attacks in a wide range of patients. On the other hand, severely underestimating statin side effects could result in the discounting of important symptoms and the suffering of patients. Abramson’s misinterpretation of statin side effect data was pointed out by readers of the BMJ soon after the article published, and it prompted an inquiry by the journal. After re-evaluating the data and discussing the issue with Abramson and colleagues, the journal issued a correction in which it clarified the misrepresentation of the Zhang paper.

Fiona Godlee, the editor-in-chief of the BMJ also wrote an editorial explaining the decision to issue a correction regarding the question of side effects and that there was not sufficient cause to retract the whole paper since the other points made by Abramson and colleagues – the lack of benefit in low risk patients – might still hold true. Instead, Godlee recognized the inherent bias of a journal’s editor when it comes to deciding on whether or not to retract a paper. Every retraction of a peer reviewed scholarly paper is somewhat of an embarrassment to the authors of the paper as well as the journal because it suggests that the peer review process failed to identify one or more major flaws. In a commendable move, the journal appointed a multidisciplinary review panel which includes leading cardiovascular epidemiologists. This panel will review the Abramson paper as well as another BMJ paper which had also cited the inaccurately high frequency of statin side effects, investigate the peer review process that failed to identify the erroneous claims and provide recommendations regarding the ultimate fate of the papers.

 

Reviewing peer review

Why didn’t the peer reviewers who evaluated Abramson’s article catch the error prior to its publication? We can only speculate as to why such a major error was not identified by the peer reviewers. One has to bear in mind that “peer review” for academic research journals is just that – a review. In most cases, peer reviewers do not have access to the original data and cannot check the veracity or replicability of analyses and experiments. For most journals, peer review is conducted on a voluntary (unpaid) basis by two to four expert reviewers who routinely spend multiple hours analyzing the appropriateness of the experimental design, methods, presentation of results and conclusions of a submitted manuscript. The reviewers operate under the assumption that the authors of the manuscript are professional and honest in terms of how they present the data and describe their scientific methodology.

In the case of Abramson and colleagues, the correction issued by the BMJ refers not to Abramson’s own analysis but to the misreading of another group’s research. Biomedical research papers often cite 30 or 40 studies, and it is unrealistic to expect that peer reviewers read all the cited papers and ensure that they are being properly cited and interpreted. If this were the expectation, few peer reviewers would agree to serve as volunteer reviewers since they would have hardly any time left to conduct their own research. However, in this particular case, most peer reviewers familiar with statins and the controversies surrounding their side effects should have expressed concerns regarding the extraordinarily high figure of 18% cited by Abramson and colleagues. Hopefully, the review panel will identify the reasons for the failure of BMJ’s peer review system and point out ways to improve it.

 

To err is human, to study errors is science

All researchers make mistakes, simply because they are human. It is impossible to eliminate all errors in any endeavor that involves humans, but we can construct safeguards that help us reduce the occurrence and magnitude of our errors. Overt fraud and misconduct are rare causes of errors in research, but their effects on any given research field can be devastating. One of the most notorious occurrences of research fraud is the case of the Dutch psychologist Diederik Stapel who published numerous papers based on blatant fabrication of data – showing ‘results’ of experiments on non-existent study subjects. The field of cell therapy in cardiovascular disease recently experienced a major setback when a university review of studies headed by the German cardiologist Bodo Strauer found evidence of scientific misconduct. The significant discrepancies and irregularities in Strauer’s studies have now lead to wide-ranging skepticism about the efficacy of using bone marrow cell infusions to treat heart disease.

 

It is difficult to obtain precise numbers to quantify the actual extent of severe research misconduct and fraud since it may go undetected. Even when such cases are brought to the attention of the academic leadership, the involved committees and administrators may decide to keep their findings confidential and not disclose them to the public. However, most researchers working in academic research environments would probably agree that these are rare occurrences. A far more likely source of errors in research is the cognitive bias of the researchers. Researchers who believe in certain hypotheses and ideas are prone to interpreting data in a manner most likely to support their preconceived notions. For example, it is likely that a researcher opposed to statin usage will interpret data on side effects of statins differently than a researcher who supports statin usage. While Abramson may have been biased in the interpretation of the data generated by Zhang and colleagues, the field of cardiovascular regeneration is currently grappling in what appears to be a case of biased interpretation of one’s own data. An institutional review by Harvard Medical School and Brigham and Women’s Hospital recently determined that the work of Piero Anversa, one of the world’s most widely cited stem cell researchers, was significantly compromised and warranted a retraction. His group had reported that the adult human heart exhibited an amazing regenerative potential, suggesting that roughly every 8 to 9 years the adult human heart replaces its entire collective of beating heart cells (a 7% – 19% yearly turnover of beating heart cells). These findings were in sharp contrast to a prior study which had found only a minimal turnover of beating heart cells (1% or less per year) in adult humans. Anversa’s finding was also at odds with the observations of clinical cardiologists who rarely observe a near-miraculous recovery of heart function in patients with severe heart disease. One possible explanation for the huge discrepancy between the prior research and Anversa’s studies was that Anversa and his colleagues had not taken into account the possibility of contaminations that could have falsely elevated the cell regeneration counts.

 

Improving the quality of research: peer review and more

Despite the fact that researchers are prone to make errors due to inherent biases does not mean we should simply throw our hands up in the air, say “Mistakes happen!” and let matters rest. High quality science is characterized by its willingness to correct itself, and this includes improving methods to detect and correct scientific errors early on so that we can limit their detrimental impact. The realization that lack of reproducibility of peer-reviewed scientific papers is becoming a major problem for many areas of research such as psychology, stem cell research and cancer biology has prompted calls for better ways to track reproducibility and errors in science.

One important new paradigm that is being discussed to improve the quality of scholar papers is the role of post-publication peer evaluation. Instead of viewing the publication of a peer-reviewed research paper as an endpoint, post publication peer evaluation invites fellow scientists to continue commenting on the quality and accuracy of the published research even after its publication and to engage the authors in this process. Traditional peer review relies on just a handful of reviewers who decide about the fate of a manuscript, but post publication peer evaluation opens up the debate to hundreds or even thousands of readers which may be able to detect errors that could not be identified by the small number of traditional peer reviewers prior to publication. It is also becoming apparent that science journalists and science writers can play an important role in the post-publication evaluation of published research papers by investigating and communicating research flaws identified in research papers. In addition to helping dismantle the Science Mystique, critical science journalism can help ensure that corrections, retractions or other major concerns about the validity of scientific findings are communicated to a broad non-specialist audience.

In addition to these ongoing efforts to reduce errors in science by improving the evaluation of scientific papers, it may also be useful to consider new pro-active initiatives which focus on how researchers perform and design experiments. As the head of a research group at an American university, I have to take mandatory courses (in some cases on an annual basis) informing me about laboratory hazards, ethics of animal experimentation or the ethics of how to conduct human studies. However, there are no mandatory courses helping us identify our own research biases or how to minimize their impact on the interpretation of our data. There is an underlying assumption that if you are no longer a trainee, you probably know how to perform and interpret scientific experiments. I would argue that it does not hurt to remind scientists regularly – no matter how junior or senior- that they can become victims of their biases. We have to learn to continuously re-evaluate how we conduct science and to be humble enough to listen to our colleagues, especially when they disagree with us.

 

Note: A shorter version of this article was first published at The Conversation with excellent editorial input provided by Jo Adetunji.

 

Abramson, J., Rosenberg, H., Jewell, N., & Wright, J. (2013). Should people at low risk of cardiovascular disease take a statin? BMJ, 347 (oct22 3) DOI: 10.1136/bmj.f6123

Advertisement

Growing Skepticism about the Stem Cell Acid Trip

In January 2014, the two papers “Stimulus-triggered fate conversion of somatic cells into pluripotency” and “Bidirectional developmental potential in reprogrammed cells with acquired pluripotency” published in the journal Nature by Haruko Obokata and colleagues took the world of stem cell research by surprise.

Since Shinya Yamanaka’s landmark discovery that adult skin cells could be reprogrammed into embryonic-like induced pluripotent stem cells (iPSCs) by introducing selected embryonic genes into adult cells, laboratories all over the world have been using modifications of the “Yamanaka method” to create their own stem cell lines. The original Yamanaka method published in 2006 used a virus which integrated into the genome of the adult cell to introduce the necessary genes. Any introduction of genetic material into a cell carries the risk of causing genetic aberrancies that could lead to complications, especially if the newly generated stem cells are intended for therapeutic usage in patients.

Researchers have therefore tried to modify the “Yamanaka method” and reduce the risk of genetic aberrations by either using genetic tools to remove the introduced genes once the cells are fully reprogrammed to a stem cell state, introducing genes without non-integrating viruses or by using complex cocktails of chemicals and growth factors in order to generate stem cells without the introduction of any genes into the adult cells.

The papers by Obokata and colleagues at the RIKEN center in Kobe, Japan use a far more simple method to reprogram adult cells. Instead of introducing foreign genes, they suggest that one can expose adult mouse cells to a severe stress such as an acidic solution. The cells which survive acid-dipping adventure (25 minutes in a solution with pH 5.7) activate their endogenous dormant embryonic genes by an unknown mechanism. The researchers then show that these activated cells take on properties of embryonic stem cells or iPSCs if they are maintained in a stem cell culture medium and treated with the necessary growth factors. Once the cells reach the stem cell state, they can then be converted into cells of any desired tissue, both in a culture dish as well as in a developing mouse embryo. Many of the experiments in the papers were performed by starting out with adult mouse lymphocytes, but the researchers also found that mouse skin fibroblasts and other cells could also be successfully converted into an embryonic-like state using the acid stress.

My first reaction was incredulity. How could such a simple and yet noxious stress such as exposing cells to acid be sufficient to initiate a complex “stemness” program? Research labs have spent years fine-tuning the introduction of the embryonic genes, trying to figure out the optimal combination of genes and timing of when the genes are essential during the reprogramming process. These two papers propose that the whole business of introducing stem cell genes into adult cells was unnecessary – All You Need Is Acid.

 

This sounds too good to be true. The recent history in stem cell research has taught us that we need to be skeptical. Some of the most widely cited stem cell papers cannot be replicated. This problem is not unique to stem cell research, because other biomedical research areas such as cancer biology are also struggling with issues of replicability, but the high scientific impact of burgeoning stem cell research has forced its replicability issues into the limelight. Nowadays, whenever stem cell researchers hear about a ground-breaking new stem cell discovery, they often tend to respond with some degree of skepticism until multiple independent laboratories can confirm the results.

My second reaction was that I really liked the idea. Maybe we had never tried something as straightforward as an acid stress because we were too narrow-minded, always looking for complex ways to create stem cells instead of trying simple approaches. The stress-induction of stem cell behavior may also represent a regenerative mechanism that has been conserved by evolution. When our amphibian cousins regenerate limbs following an injury, adult tissue cells are also reprogrammed to a premature state by the stress of the injury before they start building a new limb.

The idea of stress-induced reprogramming of adult cells to an embryonic-like state also has a powerful poetic appeal, which inspired me to write the following haiku:

 

The old warrior

plunges into an acid lake

to emerge reborn.

 

(Read more about science-related haikus here)

Just because the idea of acid-induced reprogramming is so attractive does not mean that it is scientifically accurate or replicable.

A number of concerns about potential scientific misconduct in the context of the two papers have been raised and it appears that the RIKEN center is investigating these concerns. Specifically, anonymous bloggers have pointed out irregularities in the figures of the papers and that some of the images may be duplicated. We will have to wait for the results of the investigation, but even if image errors or duplications are found, this does not necessarily mean that this was intentional misconduct or fraud. Assembling manuscripts with so many images is no easy task and unintentional errors do occur. These errors are probably far more common than we think. High profile papers undergo much more scrutiny than the average peer-reviewed paper, and this is probably why we tend to uncover them more readily in such papers. For example, image duplication errors were discovered in the 2013 Cell paper on human cloning, but many researchers agreed that the errors in the 2013 Cell paper were likely due to sloppiness during the assembly of the submitted manuscript and did not constitute intentional fraud.

Irrespective of the investigation into the irregularities of figures in the two Nature papers, the key question that stem cell researchers have to now address is whether the core findings of the Obokata papers are replicable. Can adult cells – lymphocytes, skin fibroblasts or other cells – be converted into embryonic-like stem cells by an acid stress? If yes, then this will make stem cell generation far easier and it will open up a whole new field of inquiry, leading to many new exciting questions. Do human cells also respond to acid stress in the same manner as the mouse cells? How does acid stress reprogram the adult cells? Is there an acid-stress signal that directly acts on stem cell transcription factors or does the stress merely activate global epigenetic switches? Are other stressors equally effective? Does this kind of reprogramming occur in our bodies in response to an injury such as low oxygen or inflammation because these kinds of injuries can transiently create an acidic environment in our tissues?

Researchers all around the world are currently attempting to test the effect of acid exposure on the activation of stem cell genes. Paul Knoepfler’s stem cell blog is currently soliciting input from researchers trying to replicate the work. Paul makes it very clear that this is an informal exchange of ideas so that researchers can learn from each other on a “real-time” basis. It is an opportunity to find out about how colleagues are progressing without having to wait for 6-12 months for the next big stem cell meeting or the publication of a paper confirming or denying the replication of acid-induced reprogramming. Posting one’s summary of results on a blog is not as rigorous as publishing a peer-reviewed paper with all the necessary methodological details, but it can at least provide some clues as to whether some or all of the results in the controversial Obokata papers can be replicated.

If the preliminary findings of multiple labs posted on the blog indicate that lymphocytes or skin cells begin to activate their stem cell gene signature after acid stress, then we at least know that this is a project which merits further investigation and researchers will be more willing to invest valuable time and resources to conduct additional replication experiments. On the other hand, if nearly all the researchers post negative results on the blog, then it is probably not a good investment of resources to spend the next year or so trying to replicate the results.

It does not hurt to have one’s paradigms or ideas challenged by new scientific papers as long as we realize that paradigm-challenging papers need to be replicated. The Nature papers must have undergone rigorous peer review before their publication, but scientific peer review does not involve checking replicability of the results. Peer reviewers focus on assessing the internal logic, experimental design, novelty, significance and validity of the conclusions based on the presented data. The crucial step of replicability testing occurs in the post-publication phase. The post-publication exchange of results on scientific blogs by independent research labs is an opportunity to crowd-source replicability testing and thus accelerate the scientific authentication process. Irrespective of whether or not the attempts to replicate acid-induced reprogramming succeed, the willingness of the stem cell community to engage in a dialogue using scientific blogs and evaluate replicability is an important step forward.

 

Obokata H, Wakayama T, Sasai Y, Kojima K, Vacanti MP, Niwa H, Yamato M, & Vacanti CA (2014). Stimulus-triggered fate conversion of somatic cells into pluripotency. Nature, 505 (7485), 641-7 PMID: 24476887

The Scientific Peer Review Process and its Limitations

“…I can live with doubt and uncertainty and not knowing. I think it’s much more interesting to live not knowing than to have answers which might be wrong.“

Richard P Feynman, “The Pleasure of Finding Things Out”

 

It may be useful for non-specialists who are not actively involved in the scientific peer review process to get some insight into what constitutes “scientific peer review”. This blog post will give an overview of the peer review process, primarily based on my own scientific peer review experiences in biological and medical research.

The contemporary peer review process for manuscripts submitted to scientific journals usually involves the following stages:

1. Submission: The authors submit their scientific manuscript to a journal.

2. Assignment: Once the manuscript passes an initial quality control process, it is assigned to an editor, associate editor or academic editor with expertise that is at least broadly related to the focus of the manuscript.

3. Initial editorial decision: The assigned editor decides whether the manuscript is generally appropriate for the journal and category of article that it was submitted to. If the manuscript is deemed appropriate, it is sent out for review to experts in the field. If not, the manuscript is sent back to the authors and they are asked to submit it to a more appropriate journal. Occasionally, some editors may ask the authors to substantially revise the manuscript before it can even be deemed appropriate for peer review.

Many of the high profile journals, such as Nature, Cell or Science, use this stage to eliminate the bulk of submitted manuscripts. The criteria for rejecting manuscripts at this preliminary stage are rather vague and have more to do with the general goals of the journal (i.e. publishing high-impact papers that garner citations by other scientists and frequent mentions in the news media) than the scientific rigor and quality of the scientific work. Sometimes an informal nod to the editors from a renowned leader in the field via a brief email or phone call can also do the trick and help increase the likelihood of a manuscript being sent out for review.

High-profile journals also encourage a pre-submission inquiry in which authors can submit a brief description of the work instead of the complete scientific manuscript. This allows the editors to screen the potential submissions and only invite a select group of authors for a complete manuscript submission, if they are reasonably sure that the manuscript merits a full review. The precise percentages of manuscripts that are rejected at the initial state prior to the formal review or the comparative fate of manuscripts that are submitted after a pre-submission inquiry versus those submitted directly with an inquiry are very difficult to obtain.

When manuscripts are rejected prior to a review, authors usually do not receive any specific comments other than something along the lines of “your work is not appropriate for our readership” or “please consider submitting it to a subspecialty journal”. The peer review process would be far more transparent, if journals were obligated to provide accurate statistics on what percentage of manuscripts are sent out for review, what specific factors resulted in the decision on whether or not to formally review a manuscript and the acceptance rates for manuscripts that use a pre-submission inquiry versus those which do not.

4. Selection of peer reviewers: Once a manuscript has cleared the initial editorial hurdle, it is sent out for review by scientists who are chosen based on their areas of expertise. Most journals that I have either reviewed for usually require between two and four reviewers, although I have occasionally seen cases in which seven reviewers were asked to comment on a scientific manuscript. As science is becoming more interdisciplinary, scientific manuscripts increasingly span multiple areas of research and may thus require more reviewers with complementary areas of expertise.

Many journals allow the authors to suggest names for potential peer reviewers. It is generally understood that authors will not suggest active or recent collaborators as reviewers, but instead choose scientists who are most qualified. Authors are likely to suggest reviewers who will have a favorable opinion on the importance or significance of the research, but it is the editor’s decision on whether or not to accept the author suggestions or whether to select (additional) independent reviewers. A number of journals allow requests to exclude specific scientific reviewers in case the authors feel that these scientists may harbor strong biases against the submitted scientific work or against the authors.

5. Peer review reports: Once the experts accept the invitation to review a manuscript, they are usually granted 10-14 days to submit a completed report as to the significance, novelty and scientific rigor of the manuscript. This time frame is necessary because performing an in-depth peer review requires a substantial amount of time. Since peer review is conducted by scientific experts on a voluntary, unpaid basis outside of the normal work demands, peer reviewers often devote many hours of their evenings or week-ends to pour over the submitted manuscript, the figures, data tables and other supplementary data provided by the authors. A typical manuscript in the life sciences contains 4 to 8 multi-panel figures, each consisting of multiple graphs and other images. While short papers published in journals such as Nature or Science have strict limits on word counts and also restrict the number of figures in a paper to four, there are few limits on the supplemental data. I once reviewed a paper which had six multi-panel figures as part of the main manuscript, but also contained additional 10 multi-panel figures and three tables in an online “supplementary data” section!

A conscientious peer reviewer not only reads the submitted manuscript, but also tries to envision how the experiments were designed and conducted, and tries to put the findings into context of the existing literature. A six thousand word manuscript with six figures and a couple of tables represents a distillate of two or three years of research, often conducted by a team of scientists. The authors have had months or years to familiarize themselves with the experimental design, results and interpretation of the data, and some manuscripts are written in a manner that it is not easy to divine the actual intentions of the researchers. A well-written manuscript is much easier to review.

Most editors ask the reviewers to rank or grade the significance of the work, the novelty of the research as well as experimental design or approach. In addition to providing these grades or ranks, the reviewers also offer specific comments about the strengths and weaknesses of a paper. Many of the comments relate to the adequacy of the experimental design, the consistency of the results, whether the interpretations match the presented data and how these new findings relate to previously published work. These comments vary substantially from reviewer to reviewer. Some just write a handful of sentences, others write two pages of comments.

Finally, reviewers provide the editors with a confidential overall recommendation, such as reject the manuscript, return the manuscript to authors for major revisions, return manuscript to authors for minor revisions or accept the manuscript as is.

The opinions of reviewers can vary substantially, because reviewers differ in terms of their priorities for what constitutes significant research, their analytical skills, their personal biases and their threshold for what is acceptable for publication. It is not uncommon to have one reviewer outright reject a manuscript (because the manuscript would not improve much with revision) whereas another reviewer wants to accept a manuscript pending minor revisions.

 

6. Editorial decision: The editor receives the reviewer reports and has to decide upon the overall verdict on the manuscript. If the reviews are too disparate, the editor may solicit the opinion of additional peer reviewers to help with a final decision. The final decision letter contains editorial comments as well as the comments provided by the reviewers to explain why a certain decision was reached. Many journals send a copy of the decision letter to the reviewers, not only to maintain transparency in the decision making process but also to allow peer reviewers to read the reports or comments of the anonymous reviewers. This is a form of feedback and learning opportunity, which enables peer reviewers to assess whether their fellow reviewers picked up on strengths or weaknesses of the manuscript which they may have missed. If the decision is made to ask the authors for revisions, the authors are usually given a time frame during which to make the revisions and the revised manuscript then again undergoes the review process. The editor may decide to use the same reviewers or choose additional peer reviewers.

 

******

I chose to elucidate the peer review process in such detail because I want to highlight what “peer review” does and does not entail. At its core, it is just a review of the submitted data, not a validation of the data. Peer reviewers do not perform any experiments to check the accuracy or replicability of the data contained in the submitted manuscript. The assessment of the validity of the results does not occur during the peer review process, but months or years later when other scientists attempt to replicate the published paper.

In most cases, peer reviewers do not even have access to the raw data. It is therefore very difficult for a peer reviewer to discern whether or not scientists have chosen to submit truly representative data, or whether they chose the “best” data which optimally supports the conclusions of the manuscript. Much of scientific peer review is based on the honor system. If researchers claim that they have performed an experiment five times or that the results are statistically significant, the peer reviewers take the word of the researchers and base the review of the scientific results on this assumption.

Peer-reviewed research is understandably more rigorous than research which has not undergone any review process, but the review process is quite limited in its scope and prone to errors due to the subjective priorities and biases of editors and reviewers. Validation of the research occurs when independent scientists are able to replicate the published findings. Replication of scientific results is the gulf which separates peer review from peer validation.

Blind Peers: A Path To Equality In Scientific Peer Review?

There is a fundamental asymmetry that exists in contemporary peer review of scientific papers. Most scientific journals do not hide the identity of the authors of a submitted manuscript. The scientific reviewers, on the other hand, remain anonymous. Their identities are only known to the editors, who use the assessments of these scientific reviewers to help decide whether or not to accept a scientific manuscript. Even though the comments of the reviewers are usually passed along to the authors of the manuscript, the names of the reviewers are not. There is a good reason for that. Critical comments of peer reviewers can lead to a rejection of a manuscript, or cause substantial delays in its publication, sometimes requiring many months of additional work that needs to be performed by the scientists who authored the manuscript. Scientists who receive such criticisms are understandably disappointed, but in some cases this disappointment can turn into anger and could potentially even lead to retributions against the peer reviewers, if their identities were ever disclosed. The cloak of anonymity thus makes it much easier for peer reviewers to offer honest and critical assessments of the submitted manuscript.

Unfortunately, this asymmetry – the peer reviewers knowing the names of the authors but the authors not knowing the names of the peer reviewers – can create problems. Some peer reviewers may be biased either against or in favor of a manuscript merely because they recognize the names of the authors or the institutions at which the authors work. There is an expectation that peer reviewers judge a paper only based on its scientific merit, but knowledge of the authors could still consciously or subconsciously impact the assessments made by the peer reviewers. Scientific peer reviewers may be much more lenient towards manuscripts of colleagues that they have known for many years and who they consider to be their friends. The reviewers may be more critical of manuscripts submitted by rival groups with whom they have had hostile exchanges in the past or by institutions that they do not trust. A recent study observed that scientists who review applications of students exhibit a subtle gender bias that favors male students, and it may be possible that similar gender bias exists in the peer review evaluation of manuscripts.

The journals Nature Geoscience and Nature Climate Change of  the Nature Publishing Group have recently announced a new “Double-blind peer review” approach to correct this asymmetry. The journals will allow authors to remain anonymous during the peer review process. The hope is that hiding the identities of the authors could reduce bias among peer reviewers.  The journals decided to implement this approach on a trial basis following a survey, in which three-quarters of respondents were supportive of a double-blind peer review. As the announcement correctly points out, this will only work if the authors are willing to phrase their paper in a manner that does not give away their identity. Instead of writing “as we have previously described”, authors write “as has been previously described” when citing prior publications.

The editors of Nature Geoscience state:

From our experience, authors who try to guess the identity of a referee are very often wrong. It seems unlikely that referees will be any more successful when guessing the identity of authors.

I respectfully disagree with this statement. Reviewers can remain anonymous because they rarely make direct references to their own work in the review process. Authors of a scientific manuscript, on the other hand, often publish a paper in the context of their own prior work. Even if the names and addresses of the authors were hidden on the title page and even if the usage of first-person pronouns in the context of prior publications was omitted, the manuscript would likely still contain multiple references to a group’s prior work. These references as well as any mentions of an institution’s facilities or administrative committees that approve animal and human studies could potentially give away the identity of the authors. It would be much easier for reviewers to guess the identity of some of the authors than for authors to guess the identity of the reviewers.

 

But even if referees correctly identify the research group that a paper is coming from, they are much less likely to guess who the first author is. One of our motivations for setting up a double-blind trial is the possibility that female authors are subjected to tougher peer review than their male colleagues — a distinct possibility in view of evidence that subtle gender biases affect assessments of competence, appropriate salaries and other aspects of academic life (Proc. Natl Acad. Sci. USA 109, 16474–16479; 2012). If the first author is unknown, this bias will be largely removed.

 

The double-blind peer review system would definitely make it harder to guess the identity of the first author and would remove biases of reviewers associated with knowing the identity of first authors. The references to prior work would enable a reviewer to infer that the submitted manuscript was authored by the research group of the senior scientist X at the University Y, but it would be nearly impossible for the reviewer to ascertain the identity of the first authors (often postdoctoral fellows, graduate students or junior faculty members). However, based on my discussions with fellow peer reviewers, I think that it is rather rare for reviewers to have a strong bias against or in favor of first authors. The biases are usually associated with knowing the identity of the senior or lead authors.

Many scientists would agree that there is a need for reforming the peer review process and that we need to reduce biased assessments of submitted manuscripts. However, I am not convinced that increasing blindness is necessarily the best approach. In addition to the asymmetry of anonymity in contemporary peer review, there is another form of asymmetry that should be addressed: Manuscripts are eventually made public, the comments of peer reviewers usually are not made public.

This asymmetry allows some peer reviewers to be sloppy in their assessments of manuscripts. While some peer reviewers provide thoughtful and constructive criticism, others just make offhanded comments, either dismissing a manuscript for no good reason or sometimes accepting it without carefully evaluating all its strengths and weaknesses. The solution to this problem is not increasing “blindness”, but instead increasing transparency of the peer review process. The open access journal F1000Research has a post-publication review process for scientific manuscripts, in which a paper is first published and the names and assessments of the referees are openly disclosed.  The open access journal PeerJ offers an alternate approach, in which peer reviewers can choose to either disclose their names or to stay anonymous and authors can choose to disclose the comments they received during the peer review process. This “pro-choice” model would allow reviewers to remain anonymous even if the authors choose to publicly disclose the reviewer comments.

Scientific peer review can play an important role in ensuring the quality of science, if it is conducted appropriately and provides reasonably objective and constructive critiques. Constructive criticism is essential for the growth of scientific knowledge. It is important that we foster a culture of respect for criticism in science, whether it occurs during the peer review process or when science writers analyze published studies. “Double blind” is an excellent way to collect experimental data, because it reduces the bias of the experimenter, but it may not be the best way to improve peer review. When it comes to peer review and scientific criticism, we should strive for more transparency and a culture of mutual respect and dialogue.

New Directions In Scientific Peer Review

Most scientists have a need-hate relationship with scientific peer review. We know that we need some form of peer review, because it is an important quality control measure that is supposed to help prevent the publication of scientifically invalid results. However, we also tend to hate scientific peer review in its current form, because we have had many frustrating experiences with it.

We recently submitted a manuscript to a journal, where it was stuck for more than one year, undergoing multiple rounds revisions in response to requests by the editors and the reviewers, after which they finally rejected it. The reviewers did not necessarily question the validity of our results, but they wanted us to test additional cell lines, confirm many of the findings with multiple methods and identify additional mechanisms that might explain our findings so that the paper started ballooning in size. I was frustrated because I felt that there was no end in sight. There are always novel mechanisms that one has not investigated. A scientific paper is not meant to investigate every possible explanation for a phenomenon, because that would turn the paper into a never-ending saga –every new finding usually raises even more questions.

We received a definitive rejection after multiple rounds of revisions (taking more than a year), but I was actually relieved because the demands of the reviewers were becoming quite excessive. We resubmitted the manuscript to a different journal, for which we had to scale back the manuscript. The new journal had different size restrictions and some of the revisions only made sense in the context of those specific reviewer requests and did not necessarily belong in the manuscript. This new set of reviewers also made some requests for revisions, but once we had made those revisions, the manuscript was published within a matter of months.

I have also had frustrating experiences as a scientific peer reviewer. Some authors completely disregard suggestions for improving the manuscript, and it is really up to the individual editors to decide who they side with. Scientific peer review in its current form also does not involve testing for reproducibility. As reviewers, we have to accept the authors’ claims that they have conducted sufficient experiments to test the reproducibility and validity of their data. Reviewers do not check whether their own laboratory or other laboratories can replicate the results described in the manuscript. Scientific peer reviewers have to rely on the scientific integrity of the authors, even if their gut instinct tells them that these results may not be reproducible by other laboratories.

Due to these experiences, many scientists like to say that the current peer review system is “broken”, and we know that we need radical changes to make the peer review process more reliable and fair. There are two new developments in scientific peer review that sound very interesting: Portable peer review and open peer review.

Richard Van Noorden describes the concept of portable peer review that will soon be offered by a new company called Rubriq, which will conduct the scientific peer review and provide the results for a fee to the editors of the journal. Interestingly, Rubriq will also pay peer reviewers, something which is quite unusual in the current peer review system, which relies on scientists volunteering their time as peer reviewers.  The basic idea is that if journal rejects a paper after the peer review conducted by Rubriq, the comments of the reviewers would still used by the editors of the new journal as long as it also subscribes to the Rubriq service. This would cut down on the review time at the new journal, because the editors could base their decision of acceptance or rejection on the existing reviews instead of sending out the paper for another new, time consuming review. I like this idea, because it “recycles” the efforts of the first round of review and will likely streamline the review process. My only concern is that reviewers currently use different review criteria, depending on what journal they review for. When reviewing for a “high prestige” journal, reviewers tend to set a high bar for novelty and impact and their comments likely reflect this. It may not be very easy for editors to use these reviews for a very different journal. Furthermore, editors get to know their reviewers over time and pick certain reviewers that they believe will give the most appropriate reviews for a submitted manuscript. I am not sure that editors of journals would be that pleased by “farming out” this process to a third party.

The second new development is the concept of open peer review, as proposed by the new open access scientific journal PeerJ. I briefly touched on this when discussing a paper on the emotional impact of genetic testing, but I would like to expand on this, because I am very intrigued by the idea of open peer review. In this new peer review system, the scientific peer reviewers can choose to either remain anonymous or disclose their names. One would think that peer reviewers should be able to stand by their honest, constructive peer reviews so there should be no need for anonymity. On the other hand, some scientists might worry about (un)professional repercussions because some authors may be offended by the critiques. Therefore, I think it is quite reasonable that PeerJ permits anonymity of the reviewers.

The true novelty of the open review system is that the authors can choose to disclose the peer review correspondence, which includes the initial comments by the reviewers as well as their own rebuttal and revisions. I think that this is a very important and exciting development in peer review. It forces the peer reviewers to remain civil and reasonable in their comments. Even if a reviewer chooses to remain anonymous, they are probably still going to be more thoughtful in their reviews of the manuscript if they realize that potentially hundreds or thousands of other scientists could have a peek at their comments. Open peer review allows the public and the scientific community to peek behind the usually closed doors of scientific peer reviews. This provides a certain form of public accountability for the editors. They cannot just arbitrarily accept or reject manuscripts without good reasons, because by opening up the review process to the public they may have to justify their decisions based on the reviews they solicited. One good example for the civil tone and reasonable review requests and responses can be found in the review of the BRCA gene testing paper. The reviewers (one of them chooses to remain anonymous) ask many excellent questions, including questions about the demographics and educational status of the participants. The authors’ rebuttal to some of the questions was that they did not collect the data and cannot include it in the manuscript, but they also expand some of the presented data and mention caveats of their study in the revised discussion. The openness of the review process now permits the general reader to take advantage of the insights of the reviewers, such as the missing information about the educational status of the participants.

The open review system is one of the most important new advances in scientific peer review and I hope that other journals (even the more conservative, traditional and non-open access journals) will implement a similar open peer review system. This will increase accountability of reviewers and editors, and hopefully improve the efficiency and quality of scientific peer review.

Is Kindness Key to Happiness and Acceptance for Children?

The study “Kindness Counts: Prompting Prosocial Behavior in Preadolescents Boosts Peer Acceptance and Well-Being” published by Layous and colleagues in the journal PLOS One on December 26, 2012 was cited by multiple websites as proof of how important it is to teach children to be kind. NPR commented on the study in the blog post “Random Acts Of Kindness Can Make Kids More Popular“, and the study was also discussed in ScienceDaily in “Kindness Key to Happiness and Acceptance for Children“, Fox News in “No bullies: Kind kids are most popular” and the Huffington Post in “Kind Kids Are Happier And More Popular (STUDY)“.

According to most of these news reports, the design of the study was rather straightforward. Schoolchildren ages 9 to 11 in a Vancouver school district were randomly assigned to two groups for a four week intervention: Half of the children were asked to perform kind acts, while the other half were asked to keep track of pleasant places they visited. Happiness and acceptance by their peers was assessed at the beginning and the end of the four week intervention period. The children were allowed to choose the “acts of kindness” or the “pleasant places”. The “acts of kindness” group chose acts such as sharing their lunch or giving their mothers a hug. The “pleasant places” group chose to visit places such as the playground or a grandparent’s house.

At the end of the four week intervention, both groups of children showed increased signs of happiness, but the news reports differed in terms of the impact of the intervention on the acceptance of the children.

 

The NPR blog reported:

… the children who performed acts of kindness were much more likely to be accepting of their peers, naming more classmates as children they’d like to spend time with.

This would mean that the children performing the “acts of kindness” were the ones that became more accepting of others.

 

The conclusion in the Huffington Post was quite different:

 

The students were asked to report how happy they were and identify classmates they would like to work with in school activities. After four weeks, both groups said they were happier, but the kids who had performed acts of kindness reported experiencing greater acceptance from their peers  –  they were chosen most often by other students as children the other students wanted to work with.

The Huffington Post interpretation (a re-post from Livescience) was that the children performing the “acts of kindness” became more accepted by others, i.e. more popular.

 

Which of the two interpretations was the correct one? Furthermore, how significant were the improvements in happiness and acceptance?

 

I decided to read the original PLOS One paper and I was quite surprised by what I found:

The manuscript (in its published form, as of December 27, 2012) had no figures and no tables in the “Results” section. The entire “Results” section consisted of just two short paragraphs. The first paragraph described the affect and happiness scores:

 

Consistent with previous research, overall, students in both the kindness and whereabouts groups showed significant increases in positive affect (γ00 = 0.15, S.E. = 0.04, t(17) = 3.66, p<.001) and marginally significant increases in life satisfaction (γ00 = 0.09, S.E. = 0.05, t(17) = 1.73, p = .08) and happiness (γ00 = 0.11, S.E. = 0.08, t(17) = 1.50, p = .13). No significant differences were detected between the kindness and whereabouts groups on any of these variables (all ps>.18). Results of t-tests mirrored these analyses, with both groups independently demonstrating increases in positive affect, happiness, and life satisfaction (all ts>1.67, all ps<.10).

 

There are no actual values given, so it is difficult to know how big the changes are. If a starting score is 15, then a change of 1.5 is only a 10% change. On the other hand, if the starting score is 3, then a change of 1.5 represents a 50% change. The Methods section of the paper also does not describe the statistics employed to analyze the data. Just relying on arbitrary p-value thresholds is problematic, but if one were to use the infamous p-value threshold of 0.05 for significance, one can assume that there was a significant change in the affect or mood of children (p-value <0.001), a marginally significant trend of increased life satisfaction (p-value of 0.08) and no really significant change in happiness (p-value of 0.13).

It is surprising that the authors do not show the actual scores for each of the two groups. After all, one of the goals of the study was to test whether performing “acts of kindness” has a bigger impact on happiness and acceptance than the visiting “pleasant places” (“whereabouts” group). There is a generic statement “ No significant differences were detected between the kindness and whereabouts groups on any of these variables (all ps>.18).”, but what were the actual happiness and satisfaction scores for each of the groups? The next sentence is also cryptic: “Results of t-tests mirrored these analyses, with both groups independently demonstrating increases in positive affect, happiness, and life satisfaction (all ts>1.67, all ps<.10).” Does this mean that p<0.1 was the threshold of significance? Do these p-values refer to the post-intervention versus pre-intervention analysis for each tested variable in each of the two groups? If yes, why not show the actual data for both groups?

 

The second (and final) paragraph of the Results section described acceptance of the children by their peers. Children were asked who they would like to “would like to be in school activities [i.e., spend time] with’’:

 

All students increased in the raw number of peer nominations they received from classmates (γ00 = 0.68, S.E. = 0.27, t(17) = 2.37, p = .02), but those who performed kind acts (M = +1.57; SD = 1.90) increased significantly more than those who visited places (M = +0.71; SD = 2.17), γ01 = 0.83, S.E. = 0.39, t(17) = 2.10, p = .05, gaining an average of 1.5 friends. The model excluded a nonsignificant term controlling for classroom size (p = .12), which did not affect the significance of the kindness term. The effects of changes in life satisfaction, happiness, and positive affect on peer acceptance were tested in subsequent models and all found to be nonsignificant (all ps>.54). When controlling for changes in well-being, the effect of the kindness condition on peer acceptance remained significant. Hence, changes in well-being did not predict changes in peer acceptance, and the effect of performing acts of kindness on peer acceptance was over and above the effect of changes in well-being.

 

This is again just a summary of the data, and not the actual data itself. Going to “pleasant places” increased the average number of “friends” (I am not sure I would use “friend” to describe someone who nominates me as a potential partner in a school activity) by 0.71, performing “acts of kindness” increased the average number of friends by 1.57. It did answer the question that was raised by the conflicting news reports. According to the presented data, the “acts of kindness” kids were more accepted by others and there was no data on whether they also became more accepting of others. I then looked at the Methods section to understand the statistics and models used for the analysis and found that there were no details included in the paper. The Methods section just ended with the following sentences:

 

Pre-post changes in self-reports and peer nominations were analyzed using multilevel modeling to account for students’ nesting within classrooms. No baseline condition differences were found on any outcome variables. Further details about method and results are available from the first author.

 

Based on reviewing the actual paper, I am quite surprised that PLOS One accepted it for publication. There are minimal data presented in the paper, no actual baseline scores regarding peer acceptance or happiness, incomplete methods and the rather grand title of “Kindness Counts: Prompting Prosocial Behavior in Preadolescents Boosts Peer Acceptance and Well-Being” considering the marginally significant data. One is left with many unanswered questions:

1) What if kids had not been asked to perform additional “acts of kindness” or additional visits to “pleasant places” and had instead merely logged these positive activities that they usually performed as part of their routine? This would have been a very important control group.

2) Why did the authors only show brief summaries of the analyses and omit to show all of the actual affect, happiness, satisfaction and peer acceptance data?

3) Did the kids in both groups also become more accepting of their peers?

 

It is quite remarkable that going to places one likes, such as a shopping mall is just as effective pro-social behavior (performing “acts of kindness”) in terms of improving happiness and well-being. The visits to pleasant places also helped gain peer acceptance, just not quite as much as performing acts of kindness. However, the somewhat selfish sounding headline “Hanging out at the mall makes kids happier and a bit more popular” is not as attractive as the warm and fuzzy headline “Random acts of kindness can make kids more popular“. This may be the reason why the “prosocial” or “kindness” aspect of this study was emphasized so strongly by the news media.

 

In summary, the limited data in this published paper suggests that children who are asked to intentionally hang out at places they like and keep track of these for four weeks seem to become happier, similar to kids who make an effort to perform additional acts of kindness. Both groups of children gain acceptance by their peers, but the children who perform acts of kindness fare slightly better. There are no clear descriptions of the statistical methods, no actual scores for the two groups (only the changes in scores are shown) and important control groups (such as children who keep track of their positive activities, without increasing them) are missing. Therefore, definitive conclusions cannot be drawn from these limited data. Unfortunately, none of the above-mentioned news reports highlighted the weaknesses, and instead jumped on the bandwagon of interpreting this study as scientific evidence for the importance of kindness. Some of the titles of the news reports even made references to bullying, even though bullying was not at all assessed in the study.

This does not mean that we should discourage our children from being kind. On the contrary, there are many moral reasons to encourage our children to be kind, and there is no need for a scientific justification for kindness. However, if one does invoke science as a reason for kindness, it should be based on scientifically rigorous and comprehensive data.

 

Great Expectations For Scientific Publication: How Digital Publishing Is Helping Science

I recently came across a rant that lamented the advent of digital publishing, open access publishing and self-publishing in science. The rant was published in the Huffington Post as a “digital” blog post (ah, the irony), entitled “50 Shades of Grey in Scientific Publication: How Digital Publishing Is Harming Science”. It was reminiscent of the rants that might have been uttered by calligraphers who were upset about the emergence of Gutenberg’s printing press or concerns of European aristocrats in the wake of the French Revolution about whether commoners could ever govern a country. Normally I ignore rants, but this one was written by Dr. Douglas Fields, an outstanding neuroscientist and an excellent writer, who also serves as the Chief of the Nervous System Development and Plasticity Section at the National Institute of Child Health and Human Development. It is very difficult to understand how someone who is such an eminent scientist and has an extensive experience with scientific publishing would make so many bizarre statements about open access publishing.

My initial reaction was that Dr. Fields wrote it as a satirical piece, mocking the opponents of open access publishing by listing phobias and biases of interest groups that are trying to prevent free public access to the results of scientific research. Upon further reading, I realized that perhaps Dr. Fields did consider the statements in his article to be a valid critique of open access publishing and that it therefore warrants a response to point out the errors. Dr. Björn Brembs has already written one excellent response, but I think the topic is rather important and would benefit from additional responses. My problem is figuring out how to respond to a rant that is rife with so many inaccurate statements and fallacies. I will first summarize three key flaws in Dr. Fields’ reasoning and then move on to giving specific examples.

1. Conflating digital publishing, open access publishing and self-publishing

The title of Dr. Fields’ article mentions “digital publishing”, but in the article itself, the issue of “digital publishing” is conflated with “open access publishing” and “self-publishing”, even though these are very distinct entities. Digital publishing refers to the medium of publishing, and can take the form of articles or E-books which are viewed online or downloaded. Most scientific articles that I now read have been published digitally. I am very happy about this development and I do not miss the days when I had to spend hours in the library, photo-copying hundreds of scientific articles from print journals, both wasting my time and helping commit arbocide. Some of my colleagues still like to read the paper copies of journals, but most of us prefer the convenience of being able to archive thousands of scientific articles on a single USB flash drive and not have to maneuver around large stacks of paper. When it comes to books (literary, philosophical or scientific), I feel a bit differently. I derive tactile pleasure from printed books and I still find it easier to thumb through a printed book than browse an E-book. I can sympathize with concerns regarding digital publishing of books. However, when it comes to scientific papers (the focus of Dr. Fields’ article), most scientists would agree that digital publishing has made it easier for them to stay abreast of scientific developments.

Open access publishing can be defined as the publication of articles that are freely available to everyone. Readers do not need to be affiliated with any specific organizations and they do not have to pay for reading the published material. Many digitally published articles are NOT open access. Some of the digitally published journals actually charge up to $30 for reading one article, if one does not have a personal or institutional subscription to the journal. Conversely, there are printed publications which are, in a certain sense, “open access”, such as free local newspapers or flyers with grocery coupons. Anybody can obtain these for free. These examples just highlight the difference between the medium of publishing (digital versus paper) and the access to the published material (pay-for-access versus open access).

The term self-publishing, or the more derogatory term “vanity publishing’, is used when an author pays a fee and is guaranteed publication of the manuscript. The item is published either in a paper format or a digital format, and the author does not require approval of editor or peer reviewers prior to the publication. Most papers in open access scientific journals that I have come across are NOT self-publications, because they do undergo a peer review and the editors ultimately decide whether or not the manuscript should be published.

 

2. Open access publishing and rigorous peer review

 The second major flaw is that Dr. Fields assumes open access publishing somehow impairs the peer review process and results in the publications of papers without scientific rigor. Much of my own experience stems from the PLOS family of open access journals, which include PLOS One, PLOS Medicine and PLOS Biology.  The PLOS journals are among the most widely read open access journals in biomedical research. The last time I was a peer reviewer for a PLOS One manuscript, I used the same standards to assess the validity and scientific rigor of the manuscript that I use for pay-for-access journals. The PLOS editor used my review and that of the other anonymous peer reviewers to reach a decision about the manuscript. The editor requested that the authors of the manuscript make significant scientific revisions based on our reviews, similar to what I have seen in other pay-for-access journals. I never got the impression that the open access nature of the journal in any way diminished the rigorous peer review process. Most of the academic editors and reviewers of the PLOS open access journals are scientists who also routinely review manuscripts for pay-for-access journals and I have never heard of any reviewer having separate scientific review standards for open access papers. The PLOS editorial board members, that I have spoken to, have never experienced any pressure to publish a paper that was considered to be of poor scientific quality.

There are differences in terms of the criteria regarding novelty of the scientific papers that editors of open access journals may use to decide whether or not a manuscript is suitable for their journal. PLOS Medicine, PLOS Biology and the new open access journal eLife want to primarily publish ground-breaking, high-impact papers. Significant novelty and broad impact on the field of science have to be paired with high scientific rigor for these journals to consider a manuscript for publication. PLOS One, on the other hand, is more likely to publish papers that will not have such a broad impact on science, but it still requires scientific rigor and validity of the conclusions. This hierarchy of impact is not unique to open access journals. The pay-for-access journals Nature and Science also only consider manuscripts with a potentially high impact, whereas there are many other traditional pay-for-access journals that are willing to consider lower impact research findings as long as they demonstrate scientific rigor during the peer review process.

 

3. Open access publishing and corporate interests

 The third major flaw in the article is that Dr. Fields assumes open access publishing somehow plays into the hands of capitalist corporate interests. Since open access publishing does not generate any revenue from its readers, it requires authors of manuscripts to pay for the publication costs. This might be an incentive for open access publishers to publish large numbers sub-standard scientific papers, because this way they could collect the authors’ publication fees. This is a valid concern and many of my colleagues who support open access publishing are aware of this potential conflict of interest. My response is that the reputation of an open access journal would suffer if it were to publish sub-standard scientific papers. This would lead to the loss of readers as well as submissions from authors, which would not want their work to appear in a disreputable journal. Furthermore, the peer review process of open access journals provides the necessary checks and balances to prevent publication of shoddy science. It is true that open access digital publishers could decide to increase the total number of published papers per year from say 10,000 to 100,000 in order to make more money. However, the decision about the scientific validity of a paper rests with the scientific editors and not with the publishers, so increasing the total number of available publication slots should not result in the publication of poor quality science as long as the peer review process remains rigorous and independent.

One has to also look at Dr. Fields’ concerns in the context of the current pay-for-access publishing industry, which is actually run by large corporations that reap huge profits from subscription fees. One of the largest academic publishers is Elsevier, which had over 3 billion US$ in revenues and an astonishing profit margin of 37% in 2011. The truth of the matter is that the current pay-for-access model is catering to corporate greed and is impairing the free sharing of scientific results. In addition to generating revenue from subscription fees, current pay-for-access publishers often also charge fees to the authors of the manuscripts. This may explain the high profit margin. I will elucidate this using the example of the journal Blood, one of the leading pay-for-access journals in hematology and vascular biology (which happens to be my area of interest). A review of the publication costs for Blood reveals that this journal charges $62 per printed page and $620 for each color figure, as well as $105 per data supplement. Many papers that describe the immune system, blood cells or blood vessels need to show histology or immunofluorescence images, which are usually presented in color. A hypothetical average length paper of 8 pages with four color images and one data supplement would cost the author: (8x$62) + (4x$620) +$105 = $3081. In addition to collecting this fee from the authors, the journal also then collects annual subscription fees from its readers, which are either $975 for an individual in the US, $1,220 for subscribers outside theUS and altogether much higher for a site license granted to a university library. Needless to say, these subscription fees are quite high and especially difficult to pay for in underdeveloped countries.

This $3,081 fee for a typical Blood paper has to be compared with the fees of the open access PLOS journals, which charge authors of PLOS One papers a flat fee of $1,350 (no matter how many color figures or supplements are used). The more prestigious PLOS Medicine and PLOS Biology journals charge a higher flat-fee of $2,900 per published papers. What is quite remarkable is that authors submitting to PLOS from underdeveloped countries pay either no author fees or a nominal $500 fee, depending on which country they are submitting from. Due to the open access nature of the PLOS journals, anybody can access the articles without having to pay any subscription or access fees, which is especially helpful to researchers in countries with minimal financial resources for education and research. PLOS journals are published by a non-profit organization, so this may explain why they are able to offer such affordable prices to authors and free access to all readers. However, even the recently founded open access journal Scientific Reports published by the Nature Publishing Group only charges $1,350 to the authors.

Science was never meant to be conducted by the rich for the rich. The goal of scientists should be to communicate rigorous findings to as broad an audience as possible. Open access publishing is a step in the right direction, because it helps liberate the scientific enterprise from corporate interests of pay-for-access publishers that impair the broad dissemination of scientific knowledge to readers who cannot afford the high subscription fees.

 

4. Specific responses to Dr. Fields

In this section, I will just highlight some of the statements made by Dr. Fields that I disagree with and give brief responses:

 

“Scientific publication is undergoing a drastic transformation as it passes deeper into government and capitalistic control, while weakened from struggling simultaneously to cope with unprecedented transformations brought about by electronic publication.”

“The federal government has mandated that scientific research that is funded in part by federal grants be made freely available to anyone over the Internet.”

 

Scientific publication is not passing into government and capitalistic control. It has always been under capitalistic control. The government mandates do not concern the publication itself (i.e. the government does not interfere with peer review and editorial decisions), but governments are pushing for free access to the publications. The electronic publication is not a weakening, but a strengthening because it facilitates rapid sharing and communication of scientific results.

 

“In the absence of income derived from subscriptions, scientific journals must now obtain the necessary funds for publication by charging the authors directly to publish their scientific study. The cost to authors ranges from $1,000 to $3,000 or more per article. Scientists must publish several articles a year, so these costs are substantial.

The funding model fueling open-access publication is a modern rendition of the well-known “vanity” model of publication, in which the author pays to have his or her work printed. The same well-appreciated negative consequences result when applied to scientific publication. Because the income is derived from the authors rather than from readers, the incentive for the publisher is to publish as much as possible, rather than being motivated by a primary concern for quality and significance that would increase subscription by readers, libraries and institutions and thus income. In the open-access, “author-pays” financial model, the more articles that are published, the more income the publishers collect.”

Dr. Fields is referring to the charges of open access journals, but he does not mention that the authors also have to pay substantial fees to traditional pay-for-access journals to have their work published. These charges are at times even higher than those of open access journals especially when researchers use color figures, which is common in research areas that rely heavily on fluorescence imaging (see example above for the journal Blood).  It is true that more articles may generate more profits for some open access publishers, but this is where the quality of the peer review process and editorial policies of a journal need to be evaluated. The open access PLOS journals are run by a non-profit organization and have no need to generate more profits. The problem pointed out by Dr. Fields applies to all for-profit publishing – whether it is open access or traditional pay-for-access. The best remedy is to ensure that the peer review process and editorial decisions are made by people who have no conflict of interest with the financial goals of the publishers.

 

“In place of rigorous peer review and editorial oversight by the leading scientists in the field, these publishers are substituting “innovative” approaches to review submissions, or they apply no authoritative review at all. Some open-access journals ask reviewers to evaluate only whether the techniques used in the study are valid, rather than judging the significance or novelty of the findings.”

 

Open access journals such as PLOS or eLife do have rigorous peer review in place and editorial oversight by some of the leading scientists in the field. There may be some open access journals which are not peer reviewed, but there are also pay-for-access journals that publish papers with minimal or no peer review. It is true that some open access journals such as PLOS One want reviewers to focus on the scientific validity of the research instead of whether or not it the research is deemed to be significant. This has advantages and disadvantages. By focusing on the validity instead of the significance, rigorous science gets to be published, independent of whether or not the area of research is “popular”. It also allows for the publication of studies that attempt to replicate previous work, instead of only focusing on new developments. The disadvantage is that a journal may publish multiple studies that merely re-affirm established scientific concepts or work on obscure species that are of no interest to the mainstream. On balance, I think it is a positive development, because I think that current journals under-emphasize the importance of replicating biomedical research and because I think that sometimes the “insignificant” areas of research may give rise to very new concepts that the mainstream of science would have otherwise ignored.

 

“The argument is made that the loss of rigorous scrutiny and validation provided by the traditional subscription-based mechanism of scientific publication will be replaced by the success of an article in the market after it is published — it’s the “cream-will-rise-to-the-top” theory.”

 “Now when a scientist writes up new research for publication in a prestigious journal, he or she must deal with all the contradictory findings of questionable rigor and accuracy being published by these vanity-publishing, open-access journals.”

 

As mentioned above, open access journals such as the PLOS family do have rigorous scrutiny. The open access allows for an additional mechanism of scrutiny, by allowing readers all over the world to read the article, replicate it and in some cases also comment on the article in the form of a post-publication peer review.

The phrase “vanity-publishing, open access journals” is quite bizarre. There are prestigious open access journals such as PLOS Medicine, PLOS Biology and the newly emerging journal eLife and there are prestigious pay-for-access journals such as Nature or Cell. But there are also many not so prestigious pay-for-access journals that publish work of questionable rigor or significance.

 

“Similar changes are eroding literary publication as direct electronic publication by authors on the Internet has led to erotic and reportedly pornographic works like Fifty Shades of Grey and spinoffs sweeping bestsellers lists for months. The issue is not whether erotica or pornography is or should be popular; rather, one wonders what literary work might have filled those slots on the bestsellers lists if traditional mechanisms of editor-evaluated publication had been applied, which consider more than simply the potential popularity of a work in deciding what to publish.”

 

Responding to these assertions is again a daunting task. The preceding paragraph by Dr. Fields referred to “open access publishing” of scientific papers, but this paragraph now makes references to self-published erotica or pornography. I do not understand how peer-reviewed open access papers in journals such as PLOS One or PLOS Medicine are “similar” to self-published erotica or pornography. I have read published PLOS papers and I have been a peer reviewer for PLOS manuscripts prior to their publication and I can assure you that these open access scientific papers are not very erotic and not self-published. Open access scientific papers published in journals such as PLOS papers are free of charge, whereas even self-published erotica can require the payment of a fee. Erotica have been popular for as long as literature has been popular. Some of the great works of literature are erotica or have major erotic themes, so it is not clear to me why erotica and literary works are presented as being mutually exclusive. The “reportedly pornographic” phrase makes me wonder whether Dr. Fields has even read “Fifty Shades of Grey”. Neither have I, so I cannot comment on the quality of “Fifty Shades of Grey”, but I know that the book has received some pretty bad reviews. Its literary quality or lack thereof is not necessarily a function of being self-published. Many best-sellers released by established publishers are also routinely panned by literary critics. Furthermore, even famous authors such as Marcel Proust were occasionally forced to self-publish their works, because publishing houses did not think that the manuscripts were in keeping with moral standards or that they would have a high market value.

Self-publication is actually a platform to publish books that are rejected by traditional publishers who focus on profitability and marketability of books and are averse to taking risks. It is true that the loss of editorial review in self-published books can result in the publication of poor-quality books, but that is a function of poor writing and not of self-publication. Proust and others were able to produce literary masterpieces even though they used the self-publication route. Self-publication increases the volume of published books and articles and does put an increased burden on readers and reviewers to discriminate between “good” and “bad” work. This increased burden is off-set by the opportunity that self-publishing offers for innovative books that are not considered profitable or marketable by traditional publishers. However, open access scientific journals are not self-published. Therefore the discussion about self-published erotica in the context of open access scientific publications is an unnecessary and irrelevant distraction.

 

“Scientists and the public are rightfully outraged and we all suffer when flawed scientific studies are published. Even with the most rigorous review at the best journals, flawed studies sometimes slip through, such as the “discovery” of cold fusion published in Science, but it is the rarity of this lapse that makes this so sensational when it happens. With the new open-access model of author-financed publication, the “outstanding” is drowned in a flood of trivial or unsound work.”

 

We actually do not know much about how much flawed research is published in pay-for-access journals. The vast majority of retractions stem from pay-for-access journals and so far there is no evidence that open access publication is drowning out outstanding research.

 

“The logic for this government mandate is peculiar. Why do this to science? The scientific journals claim no rights to the results of publicly funded scientific research; they only seek financial compensation for the expenses required for editing, reviewing and producing the article to validate and disseminate the findings as effectively as possible.”

 

The huge profits made by academic publishers using the pay-for-access model, where they make authors pay submission fees and also generate huge revenues from subscriptions clearly contradicts that publishers are out to maximize their profits. The answer to “Why do this to science?” is simple: Science belongs to all humankind.

“One wonders how many new advances in science will never have an opportunity to take root now that scientific publication is an increasingly corporate and government business rather than the scholarly academic activity that it was for centuries.”

Dr. Fields again forgets to mention that scientific publication has been a corporate run business in the past. The government is mandating free access to scientific research results, not running a business on the side.

 

In summary, Dr. Fields article is a diatribe against both open access and digital publishing, but few, if any, of the arguments are convincing. Many scientists who support both digital publishing and open access have great expectations for how it will help improve science. We think that digital publishing saves time and allows us to invest this time into conducting and analyzing research. Open access enables us to share scientific results with thousands and perhaps millions of students and scientists all across the globe. Scientific and medical developments published in open access journals are instantaneously available to everyone, whether it is scientists in Germany, patients in the USA or biology teachers in Mali. This allows everyone to partake in the scientific enterprise either by implementing the research findings or by enabling them to intellectually contribute to science. These are the reasons why science will improve in the new era of scientific publication. Even though we have great expectations, we also know that there will be many obstacles along the way. We have to continuously reevaluate the shift to open access publishing and we need objective and constructive criticism instead of rants to ensure that the new era of scientific publications maintains or even improves the quality of science that is published.

 

Image credit: Wikimedia / Public Domain – Artwork by H.M. Brock for Charles Dickens’ novel “Great Expectations”