U.S. science officials take aim at shoddy studies

By Sharon Begley

NEW YORK (Frankfurt: HX6.F - news) , Jan 27 (Reuters) - After years in which a handful of scientists raised alarms about biomedical studies that cannot be independently confirmed, raising the possibility that much of the research literature has been compromised by widespread incompetence or even deception, top U.S. science officials are acknowledging the crisis of "irreproducibility."

"We have to take this seriously," Dr Francis Collins, director of the National Institutes of Health (NIH), the country's largest funder of basic biomedical research, told Reuters in an interview, citing the waste of time and money spent trying to build on studies whose results are a mirage or whose methods are too opaque for others to follow.

In an essay in the journal Nature published on Monday, Collins warned that "the checks and balances that once ensured scientific fidelity have been hobbled," and outlined steps NIH will take to combat the "non-replication" problem.

On Friday, President Barack Obama's Council of Advisors on Science and Technology (PCAST) will hold a meeting on reproducibility. The council is scheduled to hear from researchers who have investigated the crisis and from journal editors. Collins said high-impact journals are "are looking for papers that are groundbreaking and surprising" - the kind most likely to report results that turn out to be phantoms.

"We wanted to learn more about the issues," said geneticist Eric Lander of the Broad Institute and PCAST co-chairman. As to whether the advisers might make any recommendations, "it's too early to know whether it would make sense for PCAST to weigh in."

Also scheduled to address PCAST is Secretary of Commerce Penny Pritzker. Pharmaceutical, biotech and medical diagnostic companies have become increasingly concerned about scientific papers whose claims hold the promise of leading to new medications or diagnostic tests - but that they can't reproduce.

Scientists at biotech pioneer Amgen (Xetra: 867900 - news) , for instance, reported in 2011 that they could confirm only six of 53 landmark studies in cancer biology. Researchers at pharma giant Bayer announced in 2012 that only 14 of 67 attempts to confirm claims in oncology, women's health and cardiovascular disease succeeded. Officials at Novartis (Xetra: NOT.DE - news) and AstraZeneca (NYSE: AZN - news) told a recent cancer meeting that they encountered the same problem.

"Whether in diagnostics or pharma, you hear the same story," said Mary Lou Gantzer, chief executive officer of closely held startup BioCore Diagnostics: " 'We saw something interesting in a paper, and we couldn't reproduce it.' "

SECRET SAUCE

Replication attempts fail for one of two reasons: The original claim is wrong, or it is correct but the scientists trying to confirm it goofed.

To address the second category, NIH is encouraging journals to devote more space for scientists to detail the methodology of their experiment. That way, there is no excuse for omitting mention of a crucial step or "secret sauce."

"I wish more replication failures could be blamed on this" habit of keeping methods partly hidden, Collins said. "But we have to recognize that many results are not replicating because they're wrong," usually because of flawed methodology.

In an attempt to remedy that, NIH is developing mandatory training in experimental design for in-house scientists this year. It is also testing whether its funding decisions should be based in part on a detailed analysis of the proposed experimental design, such as whether it includes statistical and other practices that reduce the chance of false results.

Including those best practices would make research using animal models or lab-grown cells - where most replication failures occur - more like the gold standard of clinical trials with humans.

Scientists who have led the charge against irreproducibility varied in their reactions to the NIH proposals.

"It's great that they're acknowledging this is a problem," said geneticist Elizabeth Iorns, founder and CEO of Science Exchange, an online portal that matches scientists with providers of technical services for their experiments. "But their solutions are likely to have minimal impact," since there is no rigorous research showing, for instance, that the kind of training in experimental design that NIH will require reduces poor science.

NIH could have a greater impact if it funded studies of how widespread irreproducibility is in various fields to "get a grip on which have the worst problem," Iorns said, and if it funded attempts to replicate important results. "You wouldn't need a lot of money to do this."

Last year, Iorns' "Reproducibility Initiative" began to check 50 recent high-impact papers in cancer cell biology. She expects that the verdicts on all 50 will be in this year, at a cost of $1.3 million (underwritten by the Laura and John Arnold Foundation). NIH's 2014 budget is $30 billion.

C. Glenn Begley, who led the work at Amgen attempting to confirm the cancer studies, welcomed NIH's recognition of the replication crisis and the idea of making animal research adhere to the high standards of human clinical trials. "If we could do only one thing, it would be insist that all of our studies were blinded," said Begley, chief scientific officer of closely held TetraLogic Pharmaceuticals.

He also thought training young scientists in proper methodology would help. "Most investigators aren't out to dupe anyone, but they don't know what's acceptable and what's not," he said. Some scientists have told him it's fine to report the one experiment that finds a drug candidate has an effect, for instance, and not the dozens that fail to find an effect. "They get an outlier result and present it as the result," he said.

Reforms at the institutional level can go only so far, however.

"The real problem is that scientists are reluctant to speak up about studies that won't replicate because there is so much to lose," Begley said. "If I criticize you, and you review my next grant application, you might (take revenge). That's why people are afraid to say the reason they couldn't replicate a study is that it was just plain wrong."