Transparency Is Not A One-Way Mirror

An editorial in the journal Nature published on April 24, 2013 announces an important new step in the scientific peer review process for manuscripts that are being submitted to Nature and other Nature research journals. Authors of scientific manuscripts will now be required to fill out a checklist before they can submit their work to the journal. The title of the editorial, “Announcement: Reducing our irreproducibility“, reveals the goal of this new step – addressing the problem of irreproducibility that is plaguing science. During the past year, Nature and its affiliated journals have repeatedly pointed out that the poor reproducibility rate of published research findings is a major challenge for science and that we need to develop new mechanisms to fix this problem. This new checklist may be one tiny step in the right direction. Its primary focus is the statistical reliability of the results in a submitted paper and asks authors to disclose details about the statistical analyses employed, sample size calculations, blinding and randomization. Manuscripts involving animals or human subjects are also required to disclose details about the approvals by the appropriate review boards or committees.

 

            Examples of the checklist questions are:

1. How was the sample size chosen to ensure adequate power to detect a pre-specified effect size? For animal studies, include a statement about sample size estimate even if no statistical methods were used.

5. For every figure, are statistical tests justified as appropriate? Do the data meet the assumptions of the tests (e.g., normal distribution)? Is there an estimate of variation within each group of data? Is the variance similar between the groups that are being statistically compared?

 

The authors are also reminded that they have to reveal complete statistical information in the figure legends and evidence that datasets have been submitted to public repositories for 1) Protein, DNA and RNA sequences, 2) Macromolecular structures, 3) Crystallographic data for small molecules and 4) Microarray data.

It is commendable that the Nature editors have recognized the importance of addressing the reproducibility issue in science, but I doubt that this checklist will make such a big difference. The cynical or perhaps overly honest answer to how many biologists determine sample size is not by a pre-specified sample size calculation. Instead, they might just go ahead and perform some arbitrary number of experiments with a sample size of n=5 or so, and then adjust the sample size by increasing it, if the initial results are not statistically significant until they achieve the equally arbitrary and near-mystical p-value thresholds of p<0.05 or p<0.01. This checklist will remind authors of the importance of keeping track of statistical and methodological details, and disclosing them in the manuscript. Such transparency in terms of methods and analyses is sorely needed. This will make it easier for other laboratories to attempt to replicate the published paper, but it is not clear how revealing these details will affect the chances that the results are indeed reproducible. Will the editors perhaps not review a manuscript if the checklist reveals that the authors only studied one strain of mice? Will sample sizes of n=5 not be acceptable even if the p-value is <0.01?

This brings us to another crucial point in the debate about reproducibility of scientific results. Prestigious journals such as Nature rarely review manuscripts that are deemed to be of limited significance or novelty to their readership. In fact, the vast majority of manuscripts submitted to high profile journals such as Nature of Science are rejected at the editorial level without ever undergoing a thorough peer review process. On the other hand, when editors get a personal call from high profile investigators, they may be more likely to send out a paper for review, because the publication of the paper could increase the often maligned “impact factor” of the journal.

Attempts to improve transparency and reliability of published research should not only target scientists, but also target the editorial and peer review process. Instead of the sending out a rather cryptic “Sorry, your paper is not interesting enough for us to review”, shouldn’t editors also complete a checklist that documents how they reached their decision? A checklist that addresses questions such as:

Was the acceptance/rejection of this manuscript based primarily on the scientific rigor or the number of expected citations per year?

How were the anonymous reviewers of this manuscript selected? How many of the chosen reviewers had been suggested by the authors? 

Did the authors directly interact with the editors to influence their decision whether or not to send a manuscript out for review?

Transparency is not a one-way mirror. Scientists need to become more transparent, but the editorial and review process should also be more transparent.

 

Image credit: Hall of Mirrors at Versailles (Image by Myrabella – Creative Commons License via Wikimedia)

Bone Marrow Cell Infusions Do NOT Improve Cardiac Function After Heart Attack

For over a decade, cardiologists have been conducting trials in patients using cells extracted from the bone marrow and infusing them into the blood vessels of the heart in patients who have suffered a heart attack. This type of a procedure is not without risks. It involves multiple invasive procedures in patients who are already quite ill, because they are experiencing a major heart attack:

1) Patients with a major heart attack (also referred to as ST-elevation Myocardial Infarction or STEMI) usually undergo an immediate angiogram of the heart to treat the blockage that is causing the heart attack by impeding the blood flow. This is the standard of care for heart attack patients in the developed world.

2) Patients enrolled in an experimental cell therapy trial are then brought back for a second procedure during which bone marrow is extracted with a needle under local anesthesia.

3) The research patients then undergo another angiogram of the heart using a catheter which allows for the infusion of bone marrow cells into the heart.

The hope is that the stem cells contained within the bone marrow are able to help regenerate the heart, either by turning into heart cells (cardiomyocytes), blood vessel cells (endothelial cells) or releasing factors that protect the heart and prevent the formation of a large scar. Unfortunately, there is very limited scientific evidence that bone marrow stem cells can actually turn into functional heart cells. The trials that have been conducted so far have yielded mixed results – some show that infusing the bone marrow cells indeed improves heart function, others show that patients who just receive the standard therapy with cell infusions do just as well. Most of the trials have been quite small – often studying only 10-50 patients.

The SWISS-AMI cell therapy trial, published online on April 17, 2013 in the world’s leading cardiovascular research journal Circulation, addressed this question in a randomized, controlled trial, which enrolled 200 patients who had suffered a major heart attack. The published paper is entitled “Intracoronary Injection of Bone Marrow Derived Mononuclear Cells, Early or Late after Acute Myocardial Infarction: Effects on Global Left Ventricular Function” and was conducted in Switzerland.

The researchers assigned the patients to three groups: a) Standard heart attack treatment, b) Standard heart attack treatment and infusion of bone marrow cells 5-7 days after the heart attack or c) Standard heart attack treatment and infusion of bone marrow cells 3-4 weeks after the heart attack. They assessed heart function four months later using cardiac magnetic resonance imaging, one of the best tools available to determine heart function. The results were rather disappointing: Neither of the two cell treatment groups showed any improvement in their cardiac function.

This trial had some important limitations: Even though this study enrolled 200 patients and was thus larger than most other cell therapy trials for heart attack patients, it is still a rather small study when compared to other cardiovascular studies, which routinely enroll thousands of patients. Furthermore, this study only assessed heart function after four months and it is possible that if they had waited longer, they might have seen some benefit of the cell therapy. Despite these limitations, the trial will dampen the general enthusiasm for injecting bone marrow cells into heart attack patients.

Is this study a set-back for cardiac stem cell treatments? Not really. As the authors reveal in their data analysis, most of the cells contained in the bone marrow preparation that they used for the infusion were plain old white blood cells and NOT stem cells. Actually, only 1% of the infused cells were hematopoietic stem cells (stem cells that give rise to blood cells) and there was an undisclosed percentage of other stem cell types (such as mesenchymal stem cells) contained in the infused bone marrow extract. As I point out in the accompanying editorial “Bone Marrow Tinctures for Cardiovascular Disease: Lost in Translation“, using such a mixture of poorly defined cells is ill-suited to promote cardiac regeneration or repair. Therefore, this important study is not a set-back for cardiac stem cell therapy, but a well-deserved setback for injections of undefined cells, most of which are not true stem cells!

Even if the majority of infused cells had been stem cells, there is no guarantee that merely infusing them into the heart would necessarily result in the formation of new heart tissue. Regenerating heart tissue from adult stem cells requires priming or directing stem cells towards becoming heart cells and ensuring that the cells can attach and integrate into the heart, not just infusing or injecting them into the heart.

It is commendable that the journal published this negative study, because too many treatments are being marketed as “stem cell therapies” without clarifying whether the injected cells are truly efficacious. Hopefully, the results of this trial will lead to more caution when rushing to perform “stem cell treatments” in patients without carefully defining the scientific characteristics and therapeutic potential of the cells that are being used.

 

Link to the original paper:  “Intracoronary Injection of Bone Marrow Derived Mononuclear Cells, Early or Late after Acute Myocardial Infarction: Effects on Global Left Ventricular Function”

Link to the editorial: “Bone Marrow Tinctures for Cardiovascular Disease: Lost in Translation”

Image credit: Surgeon extracting bone marrow from a patient (Public Domain image via Wikimedia)

Surder, D., Manka, R., Lo Cicero, V., Moccetti, T., Rufibach, K., Soncin, S., Turchetto, L., Radrizzani, M., Astori, G., Schwitter, J., Erne, P., Zuber, M., Auf der Maur, C., Jamshidi, P., Gaemperli, O., Windecker, S., Moschovitis, A., Wahl, A., Buhler, I., Wyss, C., Kozerke, S., Landmesser, U., Luscher, T., & Corti, R. (2013). Intracoronary Injection of Bone Marrow Derived Mononuclear Cells, Early or Late after Acute Myocardial Infarction: Effects on Global Left Ventricular Function Four months results of the SWISS-AMI trial Circulation DOI: 10.1161/CIRCULATIONAHA.112.001035

Rehman, J. (2013). Bone Marrow Tinctures for Cardiovascular Disease: Lost in Translation Circulation DOI: 10.1161/CIRCULATIONAHA.113.002775

New Directions In Scientific Peer Review

Most scientists have a need-hate relationship with scientific peer review. We know that we need some form of peer review, because it is an important quality control measure that is supposed to help prevent the publication of scientifically invalid results. However, we also tend to hate scientific peer review in its current form, because we have had many frustrating experiences with it.

We recently submitted a manuscript to a journal, where it was stuck for more than one year, undergoing multiple rounds revisions in response to requests by the editors and the reviewers, after which they finally rejected it. The reviewers did not necessarily question the validity of our results, but they wanted us to test additional cell lines, confirm many of the findings with multiple methods and identify additional mechanisms that might explain our findings so that the paper started ballooning in size. I was frustrated because I felt that there was no end in sight. There are always novel mechanisms that one has not investigated. A scientific paper is not meant to investigate every possible explanation for a phenomenon, because that would turn the paper into a never-ending saga –every new finding usually raises even more questions.

We received a definitive rejection after multiple rounds of revisions (taking more than a year), but I was actually relieved because the demands of the reviewers were becoming quite excessive. We resubmitted the manuscript to a different journal, for which we had to scale back the manuscript. The new journal had different size restrictions and some of the revisions only made sense in the context of those specific reviewer requests and did not necessarily belong in the manuscript. This new set of reviewers also made some requests for revisions, but once we had made those revisions, the manuscript was published within a matter of months.

I have also had frustrating experiences as a scientific peer reviewer. Some authors completely disregard suggestions for improving the manuscript, and it is really up to the individual editors to decide who they side with. Scientific peer review in its current form also does not involve testing for reproducibility. As reviewers, we have to accept the authors’ claims that they have conducted sufficient experiments to test the reproducibility and validity of their data. Reviewers do not check whether their own laboratory or other laboratories can replicate the results described in the manuscript. Scientific peer reviewers have to rely on the scientific integrity of the authors, even if their gut instinct tells them that these results may not be reproducible by other laboratories.

Due to these experiences, many scientists like to say that the current peer review system is “broken”, and we know that we need radical changes to make the peer review process more reliable and fair. There are two new developments in scientific peer review that sound very interesting: Portable peer review and open peer review.

Richard Van Noorden describes the concept of portable peer review that will soon be offered by a new company called Rubriq, which will conduct the scientific peer review and provide the results for a fee to the editors of the journal. Interestingly, Rubriq will also pay peer reviewers, something which is quite unusual in the current peer review system, which relies on scientists volunteering their time as peer reviewers.  The basic idea is that if journal rejects a paper after the peer review conducted by Rubriq, the comments of the reviewers would still used by the editors of the new journal as long as it also subscribes to the Rubriq service. This would cut down on the review time at the new journal, because the editors could base their decision of acceptance or rejection on the existing reviews instead of sending out the paper for another new, time consuming review. I like this idea, because it “recycles” the efforts of the first round of review and will likely streamline the review process. My only concern is that reviewers currently use different review criteria, depending on what journal they review for. When reviewing for a “high prestige” journal, reviewers tend to set a high bar for novelty and impact and their comments likely reflect this. It may not be very easy for editors to use these reviews for a very different journal. Furthermore, editors get to know their reviewers over time and pick certain reviewers that they believe will give the most appropriate reviews for a submitted manuscript. I am not sure that editors of journals would be that pleased by “farming out” this process to a third party.

The second new development is the concept of open peer review, as proposed by the new open access scientific journal PeerJ. I briefly touched on this when discussing a paper on the emotional impact of genetic testing, but I would like to expand on this, because I am very intrigued by the idea of open peer review. In this new peer review system, the scientific peer reviewers can choose to either remain anonymous or disclose their names. One would think that peer reviewers should be able to stand by their honest, constructive peer reviews so there should be no need for anonymity. On the other hand, some scientists might worry about (un)professional repercussions because some authors may be offended by the critiques. Therefore, I think it is quite reasonable that PeerJ permits anonymity of the reviewers.

The true novelty of the open review system is that the authors can choose to disclose the peer review correspondence, which includes the initial comments by the reviewers as well as their own rebuttal and revisions. I think that this is a very important and exciting development in peer review. It forces the peer reviewers to remain civil and reasonable in their comments. Even if a reviewer chooses to remain anonymous, they are probably still going to be more thoughtful in their reviews of the manuscript if they realize that potentially hundreds or thousands of other scientists could have a peek at their comments. Open peer review allows the public and the scientific community to peek behind the usually closed doors of scientific peer reviews. This provides a certain form of public accountability for the editors. They cannot just arbitrarily accept or reject manuscripts without good reasons, because by opening up the review process to the public they may have to justify their decisions based on the reviews they solicited. One good example for the civil tone and reasonable review requests and responses can be found in the review of the BRCA gene testing paper. The reviewers (one of them chooses to remain anonymous) ask many excellent questions, including questions about the demographics and educational status of the participants. The authors’ rebuttal to some of the questions was that they did not collect the data and cannot include it in the manuscript, but they also expand some of the presented data and mention caveats of their study in the revised discussion. The openness of the review process now permits the general reader to take advantage of the insights of the reviewers, such as the missing information about the educational status of the participants.

The open review system is one of the most important new advances in scientific peer review and I hope that other journals (even the more conservative, traditional and non-open access journals) will implement a similar open peer review system. This will increase accountability of reviewers and editors, and hopefully improve the efficiency and quality of scientific peer review.

Some Highlights of the Live Chat: “Are We Doing Science the Right Way?”

On February 7, 2013, ScienceNOW organized a Live Chat with the microbiologists Ferric Fang and Arturo Casadevall that was moderated by the Science staff writer Jennifer Couzin-Frankel and discussed a very broad range of topics related to how we currently conduct science. For those who could not participate in the Live Chat, I will summarize some key comments made by Fang and Casadevall, Couzin-Frankel or other commenters.

 

I have grouped the comments into key themes and also added some of my own thoughts.

 

1. Introduction to the goals of the Live Chat:

Jennifer Couzin-Frankel: …..For several years (at least) researchers have worried about where their profession is heading. As much as most of them love working in the lab, they’re also facing sometimes extreme pressure to land grants and publish hot papers. And surveys have shown that a subset are even bending or breaking the rules to accomplish that.….With us today are two guests who are studying the “science of science” together, and considering how to nurture discovery and reduce misconduct…

 

Pressure to publish, the difficulties to obtain grant funding, scientific misconduct – these are all topics that should be of interest to all of us who are actively engaged in science.

 

2. Science funding:

Ferric Fang: ….the way in which science is funded has a profound effect on how and what science is done. Paula Stephan has recently written an excellent book on this subject called “How Economics Shapes Science.”

Ferric Fang: Many are understandably reluctant to ask for more funding given the global recession and halting recovery. But I believe a persuasive economic case can be made for greater investment in R&D paying off in the long run. Paula Stephan notes that the U.S. spends twice as much on beer as on science each year.

 

These are great points. I often get the sense that federal funding for science and education is portrayed as an unnecessary luxury, charity or a form of waste. We have to remind people that investments in science and education are a very important investment with long-term returns.

 

3. Reproducibility and the self-correcting nature of science:

Arturo Casadevall: Is science self-correcting? Yes and No. In areas where there is a lot of interest in a subject experiments will be repeated and bad science will be ferreted out. However, until someone sets out to repeat an experiment we do not know whether it is reproducible. We do not know what percentage of the literature is right because no one has ever done a systematic study to see what fraction is reproducible.

 

I think that the reproducibility crisis is one of the biggest challenges for contemporary science. Thousands of scientific papers are published every day, and only a tiny fraction of them will ever be tested for reproducibility. There is minimal funding for attempting to replicate published data and also very little incentive for scientists, because even if they are able to replicate the published work, they will have a hard time publishing a confirmatory study. The lack of attempts to replicate scientific data creates a lot of uncertainty, because we do not really know, how much of the published data is truly valid.

 

Comment From David R Van Houten: …The absence of these weekly [lab] meetings was the single biggest factor allowing for the data fabrication and falsification that I observed 20 years ago as a PhD student. I pushed to get these meetings organized, and when they did occur, it made it easier to get the offender to stop, and easier to “salvage” original data…

 

I agree that regular lab meetings and more supervision by senior researchers and principal investigators can help contain and prevent data fabrication and falsification. However, overt data fabrication and fraud are probably not as common as “data fudging”, where experiments or data points are conveniently ignored because they do not fit the desired model. This kind of “data fudging” is not just a problem of junior scientists, but also occurs with senior scientists.

 

Ferric Fang: Peer review plays an important role in self-correction of science but as nearly everyone recognizes, it is not perfect. Mechanisms of post-publication review to address the problems are very important– these include errata, retractions, correspondences, follow up publications, and nowadays, public discussion on blogs and other websites.

 

I am glad that Fang (who is an editor-in-chief of an academic journal) recognizes the importance of post-publication review, and mentions blog discussions as one such form of post publication review.

 

4. Are salaries of scientists too low?

Comment From Shabbir: When an hedge fund manager makes 100 times more than a theoretical physicist, how can we expect the bright minds to go to science?

 

I agree that academic salaries for scientists are on the lower side, especially when compared with the salary that one can make in the private industry. However, I do not think that obscene salaries of hedge fund managers are the correct comparison. If the US wants to attract and retain excellent scientists, raising their salaries is definitely important. Scientists are routinely over-worked, balancing their research work, teaching, mentoring and administrative duties and receive very inadequate compensation. I have also observed a near-cynical attitude of many elite universities, which try to portray working as a scientist as an “honor” that should not require much compensation. This kind of abuse really needs to end.

 

5. Communicating science to the public

Arturo Casadevall: … Many scientists cannot explain their work at a dinner party and keep the other guests interested. We are passionate about what we do but we are often terrible in communicating the excitement that we feel. I think this is one area where perhaps better public communicating skills are needed and maybe some attention should be given to mastering these arts in training.

 

I could not agree more. Communicating science should be part of every PhD program, postdoctoral training and an ongoing effort when a scientist becomes an independent principal investigator.

 

6. Are we focusing on quantity rather than quality in science?

Ferric Fang: …. There are now in excess of 50,000,000 scientific publications according to one estimate, and we are in danger of creating a Library of Babel in which it is impossible to find the truth buried amidst poor quality or unimportant publications. This is in part a consequence of the “publish or perish” mentality in academia. A focus on quality rather than quantity in promotion decisions might help.

 

It is correct that the amount of scientific data being generated is overwhelming, but I am not sure that there is an easy way to find the “truth”. Scientific “truth” is very dynamic and I think it is becoming more and more difficult to publish in the high impact journals. A typical paper in a high-impact journal now has anywhere between 5 and 20 supplemental figures and tables, and that same paper could have been published as two or three separate papers just a few decades ago. We now just have many more active scientists all over the world that have begun publishing in English and we all have tools that generate huge amounts of data in a matter of weeks (such as microarrays, proteomics and metabolomics). It is likely that the number of publications will continue to rise in the next years and we need to come up with an innovative system to manage scientific information. Hopefully, scientists will realize that managing and evaluating existing scientific information is just as valuable as generating new scientific datasets.

 

This was a great and inspiring discussion and I look forward to other such Live Chat events.

 

Inspired By Snake Venom

When I remember the 80s, I think of Nena’s 99 Luftballons, Duran Duran’s Wild Boys and ….snake venom. Back in those days, I used to be a typical high school science nerd. My science nerdiness interfered with my ability to socialize with non-nerds and it was characterized by an unnecessary desire to read science books and articles that I did not really understand, just so that I could show off with some fancy science terminology. I did not have much of an audience to impress, because my class-mates usually ignored me. My high school biology teacher, Herr Sperr, was the only one who had the patience to listen to me. One of the science books that I purchased was called “Gehirn und Nervensystem” (i.e. “Brain and Nervous System”), published by Spektrum der Wissenschaft, the German publisher of Scientific American. It was a collection of Scientific American articles in the field of neuroscience that had been translated into German. I was thumbing through it, looking for some new neurobiology idea or expression that I could use to impress Herr Sperr. While browsing the book, I came across the article “Der Nervenwachstumsfaktor” (originally published in Scientific American as “The Nerve-Growth Factor” in 1979) by Rita Levi-Montalcini and Pietro Calissano.

My curiosity was piqued by this article, because I did not realize that nerves had “growth factors” and because one of the authors, Rita Levi-Montalcini, had just won the Nobel Prize in the preceding year. I started reading the article and loved it, reading it over and over again. I liked the article so much, that I did not even try to show off about it and kept the newly discovered inspiration to myself. There are many reasons why I loved the article and I will just mention two of them:

1. Scientific discovery is an exciting journey, starting and ending with unanswered questions

Levi-Montalcini and Calissano started off by describing the state of knowledge and the unanswered questions in the field of developmental neurobiology and neuronal differentiation in the 1940s, when Levi-Montalcini was about to launch her career as a scientist. They commented on how the simple yet brilliant idea to test whether tumors could influence the growth of nerves sparked a whole new field of investigation. They narrated a beautiful story of scientific discovery, from postulating a “nerve growth factor” to actually isolating and sequencing it. Despite all the advances that Levi-Montalcini and her colleagues had made, the article ended with a new mystery, that the role of the nerve growth factor may be much bigger than all the researchers suspected. The nerve growth factor was able to act on cells that were not neurons and it was unclear why this was the case. By hinting at these yet to be defined roles, the article made it clear that so much more work was necessary and I felt that an invitation was being extended to the readers to participate in the future discovery.

2. Scientific tools can harbor surprises and important clues

The article mentioned one important coincidence that helped shape the progress of discovering the sequence of the nerve growth factor. To assess whether the putative nerve growth factor contained nucleic acids, Levi-Montalcini and her colleagues exposed the “soup” that was inducing the growth of nerves to snake venom. The rationale was that snake venom (by the way, the German expression “Schlangengift” sounds even more impressive than the English “snake venom”) would degrade nucleic acids and if the growth enhancing properties disappeared, it would mean that the nerve growth inducing factor contained nucleic acids. It turned out that the snake venom unexpectedly magnified the nerve growth enhancing effects, because the venom contained large quantities of the nerve growth factor itself. This unexpected finding made it much easier for the researchers to sequence the nerve growth factor, because the snake venom now provided access to a large source of the nerve growth factor and it resulted in a new mystery: Why would snake venom contain a nerve growth factor?

In the subsequent decades, as I embarked on my own career as a scientist, I often thought about this article that I read back in high school. It inspired me to become a cell biologist and many of the projects in my laboratory today focus on the effects of growth factors on blood vessels and stem cells. The article also made me think about the importance of continuously re-evaluating the tools that we use. Sometimes our tools are not as neutral or straight-forward as we think, and this lesson is just as valid today as it was half a century ago. For example, a recent paper in Cell found that the virus used for reprogramming adult cells into stem cells is not merely a tool that allows entry of the reprogramming factors, as was previously thought. The virus tool can actually activate the stem cell reprogramming itself, reminiscent of how the “snake venom” tool was able to induce nerve growth.

Rita Levi-Montalcini was one of the world’s greatest biologists and passed away on December 30, 2012. In addition to her outstanding scientific work, she was also a shining example of an activist scientist with a conscience, who fought for education and research. I never had the opportunity to meet her in person, but I was inspired by her work and I will always see her as a role model.

Image credit: Cover of the book “Gehirn und Nervensystem” by Spektrum der Wissenschaft

Is Kindness Key to Happiness and Acceptance for Children?

The study “Kindness Counts: Prompting Prosocial Behavior in Preadolescents Boosts Peer Acceptance and Well-Being” published by Layous and colleagues in the journal PLOS One on December 26, 2012 was cited by multiple websites as proof of how important it is to teach children to be kind. NPR commented on the study in the blog post “Random Acts Of Kindness Can Make Kids More Popular“, and the study was also discussed in ScienceDaily in “Kindness Key to Happiness and Acceptance for Children“, Fox News in “No bullies: Kind kids are most popular” and the Huffington Post in “Kind Kids Are Happier And More Popular (STUDY)“.

According to most of these news reports, the design of the study was rather straightforward. Schoolchildren ages 9 to 11 in a Vancouver school district were randomly assigned to two groups for a four week intervention: Half of the children were asked to perform kind acts, while the other half were asked to keep track of pleasant places they visited. Happiness and acceptance by their peers was assessed at the beginning and the end of the four week intervention period. The children were allowed to choose the “acts of kindness” or the “pleasant places”. The “acts of kindness” group chose acts such as sharing their lunch or giving their mothers a hug. The “pleasant places” group chose to visit places such as the playground or a grandparent’s house.

At the end of the four week intervention, both groups of children showed increased signs of happiness, but the news reports differed in terms of the impact of the intervention on the acceptance of the children.

 

The NPR blog reported:

… the children who performed acts of kindness were much more likely to be accepting of their peers, naming more classmates as children they’d like to spend time with.

This would mean that the children performing the “acts of kindness” were the ones that became more accepting of others.

 

The conclusion in the Huffington Post was quite different:

 

The students were asked to report how happy they were and identify classmates they would like to work with in school activities. After four weeks, both groups said they were happier, but the kids who had performed acts of kindness reported experiencing greater acceptance from their peers  –  they were chosen most often by other students as children the other students wanted to work with.

The Huffington Post interpretation (a re-post from Livescience) was that the children performing the “acts of kindness” became more accepted by others, i.e. more popular.

 

Which of the two interpretations was the correct one? Furthermore, how significant were the improvements in happiness and acceptance?

 

I decided to read the original PLOS One paper and I was quite surprised by what I found:

The manuscript (in its published form, as of December 27, 2012) had no figures and no tables in the “Results” section. The entire “Results” section consisted of just two short paragraphs. The first paragraph described the affect and happiness scores:

 

Consistent with previous research, overall, students in both the kindness and whereabouts groups showed significant increases in positive affect (γ00 = 0.15, S.E. = 0.04, t(17) = 3.66, p<.001) and marginally significant increases in life satisfaction (γ00 = 0.09, S.E. = 0.05, t(17) = 1.73, p = .08) and happiness (γ00 = 0.11, S.E. = 0.08, t(17) = 1.50, p = .13). No significant differences were detected between the kindness and whereabouts groups on any of these variables (all ps>.18). Results of t-tests mirrored these analyses, with both groups independently demonstrating increases in positive affect, happiness, and life satisfaction (all ts>1.67, all ps<.10).

 

There are no actual values given, so it is difficult to know how big the changes are. If a starting score is 15, then a change of 1.5 is only a 10% change. On the other hand, if the starting score is 3, then a change of 1.5 represents a 50% change. The Methods section of the paper also does not describe the statistics employed to analyze the data. Just relying on arbitrary p-value thresholds is problematic, but if one were to use the infamous p-value threshold of 0.05 for significance, one can assume that there was a significant change in the affect or mood of children (p-value <0.001), a marginally significant trend of increased life satisfaction (p-value of 0.08) and no really significant change in happiness (p-value of 0.13).

It is surprising that the authors do not show the actual scores for each of the two groups. After all, one of the goals of the study was to test whether performing “acts of kindness” has a bigger impact on happiness and acceptance than the visiting “pleasant places” (“whereabouts” group). There is a generic statement “ No significant differences were detected between the kindness and whereabouts groups on any of these variables (all ps>.18).”, but what were the actual happiness and satisfaction scores for each of the groups? The next sentence is also cryptic: “Results of t-tests mirrored these analyses, with both groups independently demonstrating increases in positive affect, happiness, and life satisfaction (all ts>1.67, all ps<.10).” Does this mean that p<0.1 was the threshold of significance? Do these p-values refer to the post-intervention versus pre-intervention analysis for each tested variable in each of the two groups? If yes, why not show the actual data for both groups?

 

The second (and final) paragraph of the Results section described acceptance of the children by their peers. Children were asked who they would like to “would like to be in school activities [i.e., spend time] with’’:

 

All students increased in the raw number of peer nominations they received from classmates (γ00 = 0.68, S.E. = 0.27, t(17) = 2.37, p = .02), but those who performed kind acts (M = +1.57; SD = 1.90) increased significantly more than those who visited places (M = +0.71; SD = 2.17), γ01 = 0.83, S.E. = 0.39, t(17) = 2.10, p = .05, gaining an average of 1.5 friends. The model excluded a nonsignificant term controlling for classroom size (p = .12), which did not affect the significance of the kindness term. The effects of changes in life satisfaction, happiness, and positive affect on peer acceptance were tested in subsequent models and all found to be nonsignificant (all ps>.54). When controlling for changes in well-being, the effect of the kindness condition on peer acceptance remained significant. Hence, changes in well-being did not predict changes in peer acceptance, and the effect of performing acts of kindness on peer acceptance was over and above the effect of changes in well-being.

 

This is again just a summary of the data, and not the actual data itself. Going to “pleasant places” increased the average number of “friends” (I am not sure I would use “friend” to describe someone who nominates me as a potential partner in a school activity) by 0.71, performing “acts of kindness” increased the average number of friends by 1.57. It did answer the question that was raised by the conflicting news reports. According to the presented data, the “acts of kindness” kids were more accepted by others and there was no data on whether they also became more accepting of others. I then looked at the Methods section to understand the statistics and models used for the analysis and found that there were no details included in the paper. The Methods section just ended with the following sentences:

 

Pre-post changes in self-reports and peer nominations were analyzed using multilevel modeling to account for students’ nesting within classrooms. No baseline condition differences were found on any outcome variables. Further details about method and results are available from the first author.

 

Based on reviewing the actual paper, I am quite surprised that PLOS One accepted it for publication. There are minimal data presented in the paper, no actual baseline scores regarding peer acceptance or happiness, incomplete methods and the rather grand title of “Kindness Counts: Prompting Prosocial Behavior in Preadolescents Boosts Peer Acceptance and Well-Being” considering the marginally significant data. One is left with many unanswered questions:

1) What if kids had not been asked to perform additional “acts of kindness” or additional visits to “pleasant places” and had instead merely logged these positive activities that they usually performed as part of their routine? This would have been a very important control group.

2) Why did the authors only show brief summaries of the analyses and omit to show all of the actual affect, happiness, satisfaction and peer acceptance data?

3) Did the kids in both groups also become more accepting of their peers?

 

It is quite remarkable that going to places one likes, such as a shopping mall is just as effective pro-social behavior (performing “acts of kindness”) in terms of improving happiness and well-being. The visits to pleasant places also helped gain peer acceptance, just not quite as much as performing acts of kindness. However, the somewhat selfish sounding headline “Hanging out at the mall makes kids happier and a bit more popular” is not as attractive as the warm and fuzzy headline “Random acts of kindness can make kids more popular“. This may be the reason why the “prosocial” or “kindness” aspect of this study was emphasized so strongly by the news media.

 

In summary, the limited data in this published paper suggests that children who are asked to intentionally hang out at places they like and keep track of these for four weeks seem to become happier, similar to kids who make an effort to perform additional acts of kindness. Both groups of children gain acceptance by their peers, but the children who perform acts of kindness fare slightly better. There are no clear descriptions of the statistical methods, no actual scores for the two groups (only the changes in scores are shown) and important control groups (such as children who keep track of their positive activities, without increasing them) are missing. Therefore, definitive conclusions cannot be drawn from these limited data. Unfortunately, none of the above-mentioned news reports highlighted the weaknesses, and instead jumped on the bandwagon of interpreting this study as scientific evidence for the importance of kindness. Some of the titles of the news reports even made references to bullying, even though bullying was not at all assessed in the study.

This does not mean that we should discourage our children from being kind. On the contrary, there are many moral reasons to encourage our children to be kind, and there is no need for a scientific justification for kindness. However, if one does invoke science as a reason for kindness, it should be based on scientifically rigorous and comprehensive data.

 

The PhD Route To Becoming a Science Writer

If you know that you want to become a science writer, should you even bother with obtaining a PhD in science? There is no easy answer to this question. Any answer is bound to reflect the personal biases and experiences of the person answering the question. The science writer Akshat Rathi recently made a good case for why an aspiring science writer should not pursue a PhD. I would like to offer a different perspective, which is primarily based on my work in the life sciences and may not necessarily apply to other scientific disciplines.

I think that obtaining a PhD in science a very reasonable path for an aspiring science writer, and I will list some of the “Pros” as well as the “Cons” of going the PhD route. Each aspiring science writer has to weigh the “Pros” and “Cons” carefully and reach a decision that is based on their individual circumstances and goals.

Pros: The benefits of obtaining a science PhD

 

1. Actively engaging in research gives you a first-hand experience of science

A PhD student works closely with a mentor to develop and test hypotheses, learn how to perform experiments, analyze data and reach conclusions based on the data. Scientific findings are rarely clear-cut. A significant amount of research effort is devoted to defining proper control groups, dealing with outliers and trouble-shooting experiments that have failed. Exciting findings are not always easy to replicate. A science writer who has had to actively deal with these issues may be in a better position to appreciate these intricacies and pitfalls of scientific research than someone without this first-hand experience.

 

2. PhD students are exposed to writing opportunities

All graduate students are expected to write their own PhD thesis. Many PhD programs also require that the students write academic research articles, abstracts for conferences or applications for pre-doctoral research grants. When writing these articles, PhD students usually work closely with their faculty mentors. Most articles or grant applications undergo multiple revisions until they are deemed to be ready for submission. The process of writing an initial draft and then making subsequent revisions is an excellent opportunity to improve one’s writing skills.

Most of us are not born with an innate talent for writing. To develop writing skills, the aspiring writer needs to practice and learn from critiques of one’s peers. The PhD mentor, the members of the thesis committee and other graduate students or postdoctoral fellows can provide valuable critiques during graduate school. Even though most of this feedback will likely focus on the science and not the writing, it can reveal whether or not the readers were able to clearly understand the core ideas that the student was trying to convey.

 

3. Presentation of one’s work

Most PhD programs require that students present their work at departmental seminars and at national or international conferences. Oral presentations for conferences need to be carefully crafted so that the audience learns about the background of the work, the novel findings and the implications of the research – all within the tight time constraint of a 15-20 minute time slot. A good mentor will work with PhD students to teach them how to communicate the research findings in a concise and accurate manner. Some presentations at conferences take the form of a poster, but the challenge of designing a first-rate poster is quite similar to that of a short oral presentation. One has to condense months or years of research data into a very limited space. Oral presentations as well as poster presentations are excellent opportunities to improve one’s communication skills, which are a valuable asset for any future science writer.

 

4. Peer review

Learning to perform an in-depth critical review of scientific work is an important pre-requisite for an aspiring science writer. When PhD students give presentations at departmental seminars or at conferences, they interact with a broad range of researchers, who can offer novel perspectives on the work that are distinct from what the students may have encountered in their own laboratory. Such scientific dialogue helps PhD students learn how to critically evaluate their own scientific results and realize that there can be many distinct interpretations of their data. Manuscripts or grant applications submitted by the PhD student undergo peer review by anonymous experts in the field. The reviews can be quite harsh and depressing, but they also help PhD students and their mentors identify potential flaws in their scientific work. The ability to critically evaluate scientific findings is further enhanced when PhD students participate in journal clubs to discuss published papers or when they assist their mentors in the peer review of manuscripts.

 

5. Job opportunities

Very few writers derive enough income from their writing to cover their basic needs. This is not only true for science writers, but for writers in general and it forces writers to take on jobs that help pay the bills. A PhD degree provides the aspiring science writer with a broad range of professional opportunities in academia, industry or government. After completing the PhD program, the science writer can take on such a salaried job, while building a writing portfolio and seeking out a paid position as a science writer.

 

6. Developing a scientific niche

It is not easy to be a generalist when it comes to science writing. Most successful science writers acquire in-depth knowledge in selected areas of science. This enables them to understand the technical jargon and methodologies used in that area of research and read the original scientific papers so that they do not have to rely on secondary sources for their science writing. Conducting research, writing and reviewing academic papers and attending conferences during graduate school all contribute to the development of such a scientific niche. Having such a niche is especially important when one starts out as a science writer, because it helps define the initial focus of the writing and it also provides “credentials” in the eyes of prospective employers. This does not mean that one is forever tied to this scientific niche. Science writers and scientists routinely branch out into other disciplines, once they have established themselves.

 

Cons: The disadvantages of obtaining a science PhD

 

1. Some PhD mentors abuse their graduate students

It is no secret that there are a number of PhD mentors which treat graduate students as if they were merely an additional pair of hands. Instead of being given opportunities to develop thinking and writing skills, students are sometimes forced to just produce large amounts of experimental data. 

 

2. Some of the best science writers did not obtain PhDs in science

Even though I believe that obtaining a PhD in science is a good path to becoming a science writer, I am also aware of the fact that many excellent science writers did not take this route. Instead, they focused on developing their writing skills in other venues. One such example is Steve Silberman who is a highly regarded science writer. He has written many outstanding feature articles for magazines and blog posts for his superb PLOS blog Neurotribes. Steve writes about a diverse array of topics related to neuroscience and psychology, but has also developed certain niche areas of expertise, such as autism research.

 

3. Science writer is not a career that garners much respect among academics

PhD degrees are usually obtained under the tutelage of tenure-track or tenured academics. Their natural bias is to assume that “successful” students should follow a similar career path, i.e. obtain a PhD, engage in postdoctoral research and pursue a tenure-track academic career. Unfortunately, alternate career paths, such as becoming a science writer, are not seen in a very positive light. The mentor’s narcissistic pleasure of seeing a trainee follow in one’s foot-steps is not the only reason for this. Current academic culture is characterized by a certain degree of snobbery that elevates academic research careers and looks down on alternate careers. This lack of respect for alternate careers can be very disheartening for the student. Some PhD mentors or programs may not even take on a student if he or she discloses that their ultimate goal is to become a science writer instead of pursuing a tenure-track academic career.

 

4. A day only has 24 hours

Obtaining a PhD is a full-time job. Conducting experiments, analyzing and presenting data, reading journal articles, writing chapters for the thesis and manuscripts – all of these activities are very time-consuming. It is not easy to carve out time for science writing on the side, especially if the planned science writing is not directly related to the PhD research.

 

Choosing the right environment

 

The caveats mentioned above highlight that a future science writer has to carefully choose a PhD program. The labs/mentors that publish the most papers in high-impact journals or that happen to be located in one’s favorite city may not necessarily be the ones that are best suited to prepare the student for a future career as a science writer. On the other hand, a lab that has its own research blog indicates an interest in science communication and writing. A frank discussion with a prospective mentor about the career goal of becoming a science writer will also reveal how the mentor feels about science writing and whether the mentor would be supportive of such an endeavor. The most important take home message is that the criteria one uses for choosing a PhD program have to be tailored to the career goal of becoming a science writer.

 

Image via Wikimedia Commons(Public Domain): Portrait of Dmitry Ivanovich Mendeleev wearing the Edinburgh University professor robe by Ilya Repin.