There was an interesting story today on Morning Edition about new research on potential bias in nutrition studies funded by industry. Dr. David Ludwig of Children's Hospital in Boston led a team that analyzed 206 nutritional studies published between 1999 and 2003. More than half of these studies were at least partially industry-funded (in particular, funded by purveyors of milk, fruit juices, and soft drinks).
Ludwig's analysis of this body of literature had one set of researchers look at the 206 nutritional studies just from the point of view of their scientific conclusions (with no information about the sources of funding) and another group of researchers look at them just in terms of their funding sources. What they found was that, in the aggregate, industry funding was highly correlated with the reporting of scientific results that were favorable to the funder's financial interests -- indeed, the industry-funded studies were eight times more likely to report such favorable results than were the studies that were independently funded.
It's worth noting that Ludwig's study doesn't claim conscious manipulation of the findings on the part of the researchers. Rather, it suggests bias -- almost certainly unconscious bias. And unconscious bias is pretty hard for even the most honest scientist to turn off (because it's unconscious). One of the useful things a study like Ludwig's can do is to get scientists in the habit of examining their own work for creeping bias. Am I framing the questions my experiments are trying to answer in a way that might stack the deck in favor of my funder's product? Am I enthusiastically publishing negative results as well as positive ones? Given Ludwig's findings, it's not unreasonable to ask scientists to try to set up additional experimental controls on their own potential biases.
Of course, the guy they interviewed from the National Dairy Council, senior vice president of nutrition and product innovation Dr. Greg Miller, thought Ludwig's study was flawed, but at least in the parts of the interview that were broadcast he didn't point out any specific flaws with the study. Rather, he noted that NDC demands that the researchers it funds hew closely to the scientific method, and that these studies get published in prestigious and peer reviewed scientific journals. And that's good, as far as it goes, but since we've discussed many examples of fakery (whether confessed or alleged) that have appeared in peer reviewed journals, I think it's fair to say that scientific peer review is not definitive evidence of the soundness of reported findings. Peer review cannot, with perfect accuracy, screen out the papers that are knowingly fabricated or misleading. Thus, it seems overly optimistic to point to scientific peer review as a tool that effectively irradiates industry-funded research against contamination by unconscious bias.
I didn't see the program, but there's another option. Industry is most likely to fund the studies that will produce postive results. No one is going to send a proposal to the dairy industry asking to fund research on how milk kills cute babies. And to make the difference more profound, the industry is probably going to select and fund that studies that are most likely to give positive results. If an industry has money to fund one of two high quality studies, will it be a study which is likely to agree with their goals or a study where the results are uncertain? The clear study will be funded by industry and the other by other sources which would also amplify the finding.
And does anyone (journal editors even?) know about the funding sources of the peer reviewers for their own research?
And also peer review doesn't immunize against serious accidental errors: see Software Problem...5 retractions
I agree with the creeping-bias thing- but I also think that the current high-stakes publish-or-perish atmosphere provides a strong incentive to not examine one's results too closely. Or simply to not do the extra controls (a colleague's irreproducible and biologically irrelevant data spring to mind). What's a good way to make people more inclined to do their controls? Indoctrinate them young?
chezjake,
In theory journals always ask reviewers about conflicts of interest. This might not always result in recusing oneself. For example if you are asked to review the publication of a competitor lab, you might have reason to reject. Still as an expert in the field not linked to the original lab, you are almost highly qualified to review. The opposite extreme is whether you can review articles of your former mentors. When you consider some subfields have many connections, it's impossible to prevent this from happening.
JF,
My question isn't whether they do the extra controls, but if the lack is acknowledged in the discussion. People who wait to complete all their controls as part of a single study never publish. Are industry sponsors papers less likely to acknowledge their weaknesses in addition to being positive or is that roughly the same?
I agree with bsci that this finding may well be explained by industry's choice of funding mainly studies that are likely to give positive results. And I don't think that using better or more experimental controls can offset this unbalance - since the problem lies in the choice of research question rather than necessarily in the execution of the study (each study may in itself may be flawlessly executed with no influence from (conscious or uncounscious) bias).
You don't mention that the studies which found that trans fats are bad for you were food industry funded. Unconcious bias?
scientific peer review is not definitive evidence of the soundness of reported findings
Dr. Ludwig's work is far from showing that the industry-funded results are unsound. It does not even show that there is bias, unconcious or otherwise, towards a favorable or unfavorable interpretation. There may be other factors at play - it may be that studies which find negative results are not published. That doesn't mean the positive studies are unsound or came out positive because of bias.
Actually, it was the dairy industry guy who was pointing to peer review as the way they knew that Ludwig's study had to be flawed. The implication seemed to be that peer review would surely not let through any biased studies.
It's a nice thought, but we seem to have some evidence that this is too strong a conclusion to draw about a study from the mere fact that it has been accepted for publication in a peer reviewed journal.
Dr. Ludwig's work is far from showing that the industry-funded results are unsound. It does not even show that there is bias, unconcious or otherwise, towards a favorable or unfavorable interpretation. There may be other factors at play - it may be that studies which find negative results are not published. That doesn't mean the positive studies are unsound or came out positive because of bias.
This is the bigger issue - the lack of appetite to publish negative findings, especially from studies funded in part or whole by interested business concerns. I think the notion that said concerns are only going to go forward with studies that they feel have a high probability of being positive to their products is a valid one - even more so than the studies being inherently biased.
How come nobody ever seems to worry about bias, whether unconscious or not, caused by the vast majority of science funding being tax-financed and politically controlled?