Shannon Brownlee, reviewing Rigor Mortis by Richard Harris in the Washington Monthly, writes:
More than half of the total resources invested in biomedical research are allocated to basic research, and most ideas for studies are initiated by scientists themselves. This is a good investment—but only if the studies are conducted with rigorous attention to good scientific practice. It might sound strange, but good science is all about doing everything you can to find fault with your own theory. Or, as physicist Richard Feynman once put it, the first principle of science “is that you must not fool yourself—and you are the easiest person to fool.” Much of what ails the bad research that Harris chronicles—the poor lab techniques, incorrect application of statistics, selective reporting of data, and refusal to move away from studies that have been found to be faulty—seems to result from failing to adhere to this golden rule.
In Harris’s view, much of the irreproducibility problem has been driven by two interrelated forces: the ferocious competition for funding, and the culture of academic prestige. Researchers are judged by academic institutions, and often by their peers, on the basis of the number of papers they publish in journals with a high “impact factor.” At the top of the heap are Nature, Cell, and Science, which publish the papers that are cited the most often by other researchers—which means more eyeballs on their pages. For the journals, that translates into revenue from subscriptions and advertising. By that measure, People magazine has a much bigger impact factor than, say, the Atlantic or the Washington Monthly. And, like People, the high-impact journals often seek to publish the most fashionable, glitzy, eye-popping results, even if they are likely to turn out to be dead wrong.
Then there’s what I call the bullshit factor, the hyping of every little finding, no matter how preliminary, by academic research centers, which are ballyhooed by the press, all in the name of ginning up more money for research and more prestige for the institution—which, of course, leads to more money.