Standardizing the laboratory environment may be doing science more harm than good: Removing all variability from animal experiments makes them less reproducible, rather than more, according to a study published online today in Nature Methods.

Image:Wikipedia

The study "is certainly a clear demonstration of why standardization can indeed decrease reproducibility, and I hope that from now on this idea will appear less counterintuitive in the field," Neri Kafkafi at the University of Maryland School of Medicine, who was not involved in the research, wrote in an email.

Animal researchers have generally assumed that standardizing lab conditions as much as possible provides the cleanest experimental results, and makes it easier for other labs to reproduce the findings. But recent studies have cast doubt on that view -- especially as researchers increasingly struggle with knockout mouse strains in which the phenotype they're studying varies depending on the strain of mice. Kafkafi, for example, recently proposed some tweaks to experimental design that take into account the fact that both genetics and immediate lab environment affect experimental outcomes. However, researchers have continued to vociferously debate whether the answer to the problem is to insist on still greater standardization, or to add back variability into experiments.

Hanno Wuerbel at the University of Giessen in Germany and his colleagues reanalyzed data from a previous study, published in Nature in 2004. That study had shown that enriching mouse cage environments was good practice, and didn't increase variation in how individual mice performed during behavioral tests. In the new study, the researchers wanted to use the data from those 432 mice to ask a different question: Does including mice raised under several different conditions within a single experimental analysis make the results less reproducible?

The mice in the old study were made up of three different strains, and were tested in three different labs under different cage conditions. The researchers reorganized that data in their new "virtual" experiment, mixing up those factors to examine to see if there were any effects on the results.

The researchers found no statistical difference between the "heterogenized" groups -- suggesting that mixing up the environmental factors didn't increase the variability in the results. But when mice bred from the same genetic strains and raised under the same environmental conditions were grouped together -- i.e., when the conditions were standardized -- the researchers found statistically significant differences between groups in performance on behavioral tests. "In standardized replicates we found almost 10 times as many false positives," Wuerbel told The Scientist. That suggests that standardizing all factors in an experiment increases the chance that investigators will end up with experiment-specific results, he said.

The notion that some variability is scientifically beneficial shouldn't come as a surprise, said Wuerbel. "Basically, it's a fundamental principle in a lot of science," he explained. "If you think about clinical trials, nobody would run a human clinical trial on only 18-year-old healthy white males from a particular place." In animal research, however, the reigning assumption is the more elements of a study are standardized, the better, to help other labs recreate the exact same conditions, he added. "Of course, lab animal scientists, because they are so used to mouse standardization, they have some difficulties appreciating our approach."

In an accompanying News and Views article, Richard Paylor at the Baylor College of Medicine Houston, Texas, noted that so far, there's no practical way for researchers to inject variability into their experiments. Still, he wrote, it's impossible to standardize everything, which may help explain why trying to standardize everything creates confounding effects. Mouse houses differ at different institutions, as does climate, for example.

Wuerbel, however, insists that complete standardization shouldn't be the goal. "The point is, if you could standardize everything, it would be entirely the wrong approach," said Wuerbel. "If you think about it, if standardization was perfect it would mean there was no variation. If variation is zero, then every experiment turns into a single case study."

Wuerbel and his colleagues aren't advocating willy-nilly shake-ups of experimental procedure, however. Instead, he proposes systematically varying conditions. For example, if an experiment includes 16 mice in an experimental condition and 16 in a control condition, scientists would typically house mice of the same age under identical cage conditions. Instead, Wuerbel suggested that researchers consider using cages with both an enriched environment and a standard environment, and include mice of two different ages. That would create a 2x2 factorial design with each cage containing a set of mice that varies along those parameters. By using a statistical technique called "blocking" -- in which the varying elements are grouped together -- "the variation between conditions would be calculated out," he said.

Does including such variation mean upping the number of mice used? "That's what everybody thinks, but it's not true," he said. "That's actually the nice thing about this." The group is now running a multi-lab study to work out what the best strategies for heterogenizing conditions might be.

Proponents of heterogeneity such as Wuerbel and Kafkafi stress that it's not just behavioral sciences where the approach would boost reproducibility. Strategies to systematically increase variation may also prove relevant to "more quantitative biological fields such as brain imaging and even gene expression," Kafkafi wrote.


Related stories:

Leaching plastics throw lab assays
[6th November 2008]

The trouble with animal models
[July 2007]