Drug Trial Publication Bias and the FDA

Drug companies are not publishing all the trial data that they submit to the FDA, and those trials that are published are more likely to show positive results.

Rising et al. compared all the New Drug Applications (NDAs) (the vehicle for initiating a new clinical trial) given to the FDA in 2001 and 2002 to subsequent published literature. They found that only about 3/4 of the trials were later published in journals, and those that were published were 5 times as likely to show favorable results for the drug being tested by the drug company as those that were not:

The researchers identified all the efficacy trials included in NDAs for totally new drugs that were approved by the FDA in 2001 and 2002 and searched the scientific literature for publications between July 2006 and June 2007 relating to these trials. Only three-quarters of the efficacy trials in the NDAs were published; trials with favorable outcomes were nearly five times as likely to be published as those without favorable outcomes. Although 155 primary outcomes were in both the papers and the NDAs, 41 outcomes were only in the NDAs. Conversely, 17 outcomes were only in the papers; 15 of these favored the test drug. Of the 43 primary outcomes reported in the NDAs that showed no statistically significant benefit for the test drug, only half were included in the papers; for five of the reported primary outcomes, the statistical significance differed between the NDA and the paper and generally favored the test drug in the papers. Finally, nine out of 99 conclusions differed between the NDAs and the papers; each time, the published conclusion favored the test drug.

The authors note, I think rightly, that as the outcomes of these trials weigh heavily in medical decision making, all trials -- positive or not -- should be published.

I have a couple comments about this study.

1) This is a serious issue. As a future physician, I need accurate assessments of the effectiveness of drugs combined with their comparative effectiveness to the current standard of care. If I am not getting that information, then I am not helping my patients make good decisions. Thus, I don't think it is unreasonable for the FDA to require and the drug companies to make accessible all the information about outcomes of their drugs.

2) That being said, I cannot come down too hard on the drug companies as lying bastards for this. Academic scientists are also less likely to publish their negative results. Publishing bias is a real issue whether you are talking about academia or industry. The difference here is that we are dealing with bias that directly affect patient care -- hence more aggressive policies to eliminate that bias.

3) With respect to fixes, I have to disagree with one of the authors of the study. Lisa Bero, one of the authors of the study, suggested in a Wired magazine comment on the story that the FDA should be taking over:

Bero calls for the FDA to be overhauled to run clinical studies itself, as is done by comparable agencies in Italy and Spain.

"The Italian FDA collects money from every drug company that sells drugs in Italy, pools that, and funds drug trials. They fund the sort of head-to-head drug comparisons that companies don't like to fund. And they have independent people peer-reviewing the trials. It's a great model," she said.

Listen, I don't disagree that the FDA (or NIH) need to be funding more head-to-head studies that compare the relative effectiveness of drugs in different settings. There are far too few of these large outcomes studies, and they are really, really useful when determining treatment guidelines (what type of patient benefits from what drug?). Drug companies are understandably reluctant to fund such studies lest their product end up with the short end of the stick.

I get that. We need more active research in this area, and the FDA or NIH is needed to coordinate those things.

But the suggestion that all clinical trials should be run by the FDA is just unrealistic. The FDA doesn't even have the resources for post-trial surveillance much less running all clinical trials. What such a policy would create is an environment where 1) the FDA is deciding which are the best drug candidates and 2) it takes even longer to get a drug to market. Both would dramatically reduce the number of drugs hitting the market.

Now some of you may think that is a good thing, but considering that there are a lot of conditions out there about which as a physician I can do nothing...well, I think it is a bad idea.

Alternatively, there has got to be a technical fix to this. The authors of the study mention that one of the problems with the NDA database is that its presentation of the data is often obtuse or redacted. Alright, let's fix that for starters. We are living in a Internet-based world. It seems reasonable to me to make a de-indentified database of clinical outcomes that is both accessible and complete. This way both patients and doctors can find out these results without (or pre-) publication. You might even have advocacy groups publishing summaries of the data if it is not to be published.

It seems to be that if we require this data to be reported and we make it accessible, we won't have to worry about publication bias.

Hat-tip: Wired Science

Kristin Rising, Peter Bacchetti, Lisa Bero (2008). Reporting Bias in Drug Trials Submitted to the Food and Drug Administration: Review of Publication and Presentation PLoS Medicine, 5 (11) DOI: 10.1371/journal.pmed.0050217