Sources of Scientific Bias: Both Money and Beliefs
Posted by Max Dunn Fri, 21 Dec 2007 19:03:00 GMT
When trying to discredit a scientist’s work, critics sometimes simply note: “They received funding from XYZZY corporation.” It is interesting that often nothing more is said, as if it is apparent that taking any money from a party with a vested interest will inevitably skew the results. This seems like a pretty harsh position, but there is some basis to this claim.
For instance, new research looked at drug studies and found that while the results of the study were not likely to be biased, the conclusion from those results were. In particular, they found that studies funded by a single drug company have a 55% rate of favorable results that is transformed into a 92% rate for favorable conclusions, representing a 37% gap. The gap shrinks to 21% (57% to 79%) when two or more drug companies provide support. Yet the gap vanishes entirely for studies done by non-profit institutions alone or even in conjunction with drug companies. These findings suggest a disconnect between the data that underlie the results and the interpretation or “spin” of these data that constitutes the conclusions.[1]
So it is apparent that receiving funding from a particular source that has a vested interest in the outcome will likely affect the conclusion of a study.
But are there other biases besides funding sources that can affect the results or conclusions of scientific work? What if a researcher isn’t taking any outside funding but has a passionate belief about what they are researching – will this affect their work as well?
“Confirmation bias” is a tendency to search for or interpret new information in a way that confirms one’s preconceptions.[2] While ideally scientists should be completely objective about results and conclusions, they are people too and are affected by confirmation bias.
In one study that showed scientific confirmation bias, expert reviewers were asked to peer review one of two versions of a report related to treatments of obesity. The reports were identical except that one used an orthodox treatment, and the other an unconventional one. Since the results and conclusion were the same, both versions should have received similar ratings of importance and verdicts of whether the report should be published or rejected. However, there was a significant bias in favor of the version with the orthodox treatment.[3]
Another real-life example of confirmation bias is related to historical temperature data used by Nasa’s Goddard Institute for Space Studies (GISS). In 2000, the data supplied to GISS switched to the raw version without the time-of-observation adjustment and this caused the data to jump by 0.15 deg C.[4] The graph of the data showed an obvious discontinuity and GISS should have noticed it. However since GISS tends to be pro global warming, this jump was in a direction that confirmed their beliefs. It took someone who was a global warming sceptic to finally notice this error.[5] Overlooking this error and finally catching it are each likely the result of confirmation bias, i.e. scientists tend to find what they are already looking for.
I think both funding biases and confirmation biases are bound to skew scientific research. And rather than attempting the impossible task of trying to eliminate any bias in scientific research, it is more important to accurately and fully disclose both types of bias so readers and other researchers can be aware of which direction the results and conclusions may slant. While it is fairly common today to provide information relating to the sources of funding for scientific projects, we should also require full disclosure of the researcher’s own beliefs that may cause confirmation biases to creep into their work.
[1] Some Drug Studies More Likely To Show Bias In Favor Of Funding Body Than Others
[3] A randomized controlled study of reviewer bias against an unconventional therapy