New movement to achieve scientific reproducibility

Highlighted in the Washington Post, a new movement aimed at ensuring scientific reproducibility- primarily by making data more publicly available.  Unfortunately, some amount of fraud and irreproducibility have been plaguing the current scientific system for some time (as mentioned in previous posts), and now there’s a drive to fix these issues.

And so there’s a movement afoot, and building momentum rapidly. Roughly four centuries after the invention of the scientific method, the leaders of the scientific community are recalibrating their requirements, pushing for the sharing of data and greater experimental transparency.

Top-tier journals, such as Science and Nature, have announced new guidelines for the research they publish.

“We need to go back to basics,” said Ritu Dhand, the editorial director of the Nature group of journals. “We need to train our students over what is okay and what is not okay, and not assume that they know.”

The pharmaceutical companies are part of this movement. Big Pharma has massive amounts of money at stake and wants to see more rigorous pre-clinical results from outside laboratories. The academic laboratories act as lead-generators for companies that make drugs and put them into clinical trials. Too often these leads turn out to be dead ends.

Some pharmaceutical companies are now even willing to share data with each other, a major change in policy in a competitive business.

A core scientific principle
Reproducibility is a core scientific principle. A result that can’t be reproduced is not necessarily erroneous: Perhaps there were simply variables in the experiment that no one detected or accounted for. Still, science sets high standards for itself, and if experimental results can’t be reproduced, it’s hard to know what to make of them.

“The whole point of science, the way we know something, is not that I trust Isaac Newton because I think he was a great guy. The whole point is that I can do it myself,” said Brian Nosek, the founder of a start-up in Charlottesville, Va., called the Center for Open Science. “Show me the data, show me the process, show me the method, and then if I want to, I can reproduce it.”

The reproducibility issue is closely associated with a Greek researcher, John Ioannidis, who published a paper in 2005 with the startling title “Why Most Published Research Findings Are False.”

Ioannidis, now at Stanford, has started a program to help researchers improve the reliability of their experiments. He said the surge of interest in reproducibility was in part a reflection of the explosive growth of science around the world. The Internet is a factor, too: It’s easier for researchers to see what everyone else is doing.

“We have far more papers, far more scientists working on them, and far more opportunity to see these kinds of errors and for the errors to be consequential,” Ioannidis said.
Errors can potentially emerge from a practice called “data dredging”: When an initial hypothesis doesn’t pan out, the researcher will scan the data for something that looks like a story. The researcher will see a bump in the data and think it’s significant, but the next researcher to come along won’t see it — because the bump was a statistical fluke.

“There’s an aphorism: ‘If you torture the data long enough, they will confess.’ You can always get the data to produce something that is publishable,” says the Center for Open Science’s Nosek, who is a University of Virginia professor of psychology.

His center is known among its employees as “the COS,” which is both an acronym and a homonym. They’re really talking about “the cause” — the struggle to make science more robust.

Nosek’s operation has grown from two employees in April 2013 to 53 employees today, about half of them interns, with everyone crammed into an office about a block from the downtown pedestrian mall. They spend much of their time designing software programs that let researchers share their data.

So far about 7,000 people are using that service, and the center has received commitments for $14 million in grants, with partners that include the National Science Foundation and the National Institutes of Health, Nosek said.

Another COS initiative will help researchers register their experiments in advance, telling the world exactly what they plan to do, what questions they will ask. This would avoid the data-dredging maneuver in which researchers who are disappointed go on a deep dive for something publishable.

Nosek and other reformers talk about “publication bias.” Positive results get reported, negative results ignored. Someone reading a journal article may never know about all the similar experiments that came to naught.

There’s a natural tendency to tidy up the experiment, and make the result prettier and less ambiguous, Nosek said. Call it airbrushed science.

“What is able to get published is positive, innovative, novel, and it’s really clean and beautiful. But most research in the laboratory doesn’t look like that,” Nosek says. “We are incentivized to make our research more beautiful than it is.”

Embarrassing cases
Scientific errors get a lot of publicity, but these embarrassing cases often demonstrate science at its self-correcting best.

Consider “cold fusion”: In 1989, two scientists claimed to have achieved nuclear fusion at room temperature, previously considered impossible. It was a bombshell announcement — but no one else could replicate their work. Cold fusion didn’t take off because mainstream scientists realized it wasn’t real.

A more recent case involved “arsenic life.” In 2010 a paper in Science suggested that a bacterium in Mono Lake, Calif., used arsenic instead of phosphorus in its genetic code and represented a new form of life. Rosemary Redfield, a scientist, cast doubt on the conclusion, and other researchers couldn’t replicate the finding. The consensus is that it was a misinterpretation.

In early 2014, the scientific world was rocked by a tragic case in Japan. A young scientist, Haruko Obokata, claimed to have found evidence for a phenomenon called “STAP,” stimulus-triggered acquisition of pluripotency — a way to manipulate ordinary cells to turn them into stem cells capable of growing into a variety of tissues.

But no one else could reproduce the experiment. An investigation found Obokata guilty of misconduct, and she later resigned from her institute. The journal Nature retracted the STAP papers, and then the case took a horrific turn in August, when Obokata’s mentor, the highly respected scientist Yoshiki Sasai, hanged himself.

Betsy Levy-Paluck, an associate professor of psychology and public policy at Princeton, said of the reproducibility movement, “I think it’s the future.” But there has been controversy at the laboratory level: Some researchers have complained that the reformers are going overboard.

“There are worries about there being witch hunts,” Levy-Paluck said. She said it’s frightening to think about someone discovering your mistake after publication.

Some veteran scientists have sounded a cautious note when discussing the reproducibility surge.

“Look, science is complicated, because the world is complicated,” says Eric Lander, head of the Broad Institute at MIT and co-chair of President Obama’s Council of Advisors on Science and Technology.

Lander, who played a leading role in the decoding of the human genome, says the irreproducibility problem is caused in part by the many variables that go into any experiment. To take one simple example: During his research on the genome, he and his colleagues discovered that experiments were influenced by humidity levels in the lab. They had to control for that. He said the genomics community has also tightened the standard for a “significant” result, precisely to overcome the problem of statistical flukes being mistaken for discoveries.

“Nobody tells you in advance what variables are going to matter. There’s an art to doing science,” Lander said. “Reproducibility is actually the heart of science. The fact that not everything is reproducible is not a surprise. The remarkable thing about the scientific enterprise is that we try to reproduce things, and we worry about it.”

The scientific enterprise is growing at phenomenal speed, spurred by a hunger for knowledge and an awareness that science usually delivers reliable answers about the nature of the world.

Brian Nosek offers up a stunning factoid: More than half of the scientists who have ever lived are alive today.

Though, yeah, someone ought to double-check that.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s