Effective dishonesty in pharmaceutical research revealed by new study

The file drawer effect works like this: Numerous studies are done and the results are random. But because they are random, a small number have, randomly, strong effects that are interesting and that in isolation support some interesting hypothesis. All the results that fail to confirm the interesting (or fund able) expectation are filed away .... in the file drawer. Only the results that seem to show what the researchers want to show are made public.

In areas where research is cheap and often done as part of undergraduate and graduate training, (like certain areas of psychology and experimental archaeology) it is quite possible to assemble a very large number of unpublished pilot projects and preliminary studies. This is routine and in and of itself not particularly nefarious. Nonetheless, it is still the case that ho-hum or negative results from such studies are going to be routinely ignored, while results that seem to hit on something are not. This is how science works. You observe, observe, observe, then finally you see something interesting and that is the result you jump on. You then design new studies that re-investigate the observation in a way that rules out the various biases that may exist in the background of habitual observation.

A recent study in PLoS medicine looks at a similar effect in the study of drugs for use by humans. First the bad news: The study found that an alarming percentage of studies are either never published or published quite a bit after the drug approval process is complete, and more seriously, that the studies that are published are those that include more favorable results than the ones that are ... put in the file drawer.

The good news: Even though one would hope that science would be a bit more self correcting in this regard, it wasn't, and changes in federal regulation have emerged that will reduce these shenanigans by medical and pharmaceutical researchers.

Here's an excerpt form the study's summary (emphasis added):

Before a new drug becomes available ..., its benefits and harms are carefully studied... In the most important of these trials--so-called "pivotal" clinical trials--the efficacy and safety of the new drug and of a standard treatment are compared by giving groups of patients the different treatments and measuring several predefined "outcomes." These outcomes indicate whether the new drug is more effective than the standard treatment and whether it has any other effects on the patients' health and daily life. All this information is then submitted by the sponsor of the new drug (usually a pharmaceutical company) to the government body responsible for drug approval...

[In the US] After a drug receives ... approval, information about the clinical trials supporting the ... decision are included in the FDA "Summary Basis of Approval" and/or on the drug label. In addition, some clinical trials are described in medical journals. Ideally, all the clinical information that leads to a drug's approval should be publicly available to help clinicians make informed decisions about how to treat their patients. A full-length publication in a medical journal is the primary way that clinical trial results are communicated to the scientific community and the public. Unfortunately, drug sponsors sometimes publish the results only of trials where their drug performed well; as a consequence, trials where the drug did no better than the standard treatment or where it had unwanted side effects remain unpublished. Publication bias like this provides an inaccurate picture of a drug's efficacy and safety relative to other therapies and may lead to excessive prescribing of newer, more expensive (but not necessarily more effective) treatments.

The researchers identified 90 drugs approved by the FDA between 1998 and 2000 ... they identified 909 clinical trials undertaken to support these approvals. .... Although 76% of the pivotal trials had appeared in medical journals, usually within 3 years of FDA approval, only 43% of all of the submitted trials had been published. Among all the trials, those with statistically significant results were nearly twice as likely to have been published as those without statistically significant results, and pivotal trials were three times more likely to have been published as nonpivotal trials, 5 years postapproval....

...these findings suggest that more than half the clinical trials undertaken to support drug approval remain unpublished 5 years or more after FDA approval. They also reveal selective reporting of results. For example, they show that a pivotal trial in which the new drug does no better than an old drug is less likely to be published than one where the new drug is more effective,... Importantly, these findings provide a baseline for monitoring the effects of the FDA Amendments Act 2007, which was introduced to improve the accuracy and completeness of drug trial reporting. Under this Act, all trials supporting FDA-approved drugs must be registered when they start, and the summary results of all the outcomes declared at trial registration as well as specific details about the trial protocol must be publicly posted within a year of drug approval on the US National Institutes of Health clinical trials site.

This peer reviewed paper is published in PLoS, which means you can access it and read it yourself without being special. It is here. Also, Orac at Respectful Insolence has written about this paper. Here.

Kirby Lee, Peter Bacchetti, Ida Sim, Mike Clarke (2008). Publication of Clinical Trials Supporting Successful New Drug Applications: A Literature Analysis PLoS Medicine, 5 (9) DOI: 10.1371/journal.pmed.0050191