Publication of Clinical Trials Supporting Successful New Drug Applications: A
Literature Analysis

Summary

Background:
The United States (US) Food and Drug Administration (FDA) approves new drugs based on

sponsor-submitted clinical trials. The publication status of these trials in the medical

literature and factors associated with publication have not been evaluated. We sought to

determine the proportion of trials submitted to the FDA in support of newly approved

drugs that are published in biomedical journals that a typical clinician, consumer, or

policy maker living in the US would reasonably search.

Methods and Findings:
We conducted a cohort study of trials supporting new drugs approved between 1998 and

2000, as described in FDA medical and statistical review documents and the FDA approved

drug label. We determined publication status and time from approval to full publication

in the medical literature at 2 and 5 y by searching PubMed and other databases through

01 August 2006. We then evaluated trial characteristics associated with publication. We

identified 909 trials supporting 90 approved drugs in the FDA reviews, of which

43% (394/909) were published. Among the subset of trials described in the

FDA-approved drug label and classified as “pivotal trials” for our

analysis, 76% (257/340) were published. In multivariable logistic regression

for all trials 5 y postapproval, likelihood of publication correlated with statistically

significant results (odds ratio [OR] 3.03, 95% confidence

interval [CI] 1.78–5.17); larger sample sizes (OR 1.33 per

2-fold increase in sample size, 95% CI 1.17–1.52); and pivotal

status (OR 5.31, 95% CI 3.30–8.55). In multivariable logistic

regression for only the pivotal trials 5 y postapproval, likelihood of publication

correlated with statistically significant results (OR 2.96, 95% CI

1. 24–7.06) and larger sample sizes (OR 1.47 per 2-fold increase in sample

size, 95% CI 1.15–1.88). Statistically significant results and

larger sample sizes were also predictive of publication at 2 y postapproval and in

multivariable Cox proportional models for all trials and the subset of pivotal

trials.

Conclusions:

Over half of all supporting trials for FDA-approved drugs remained unpublished ≥

5 y after approval. Pivotal trials and trials with statistically significant results and

larger sample sizes are more likely to be published. Selective reporting of trial

results exists for commonly marketed drugs. Our data provide a baseline for evaluating

publication bias as the new FDA Amendments Act comes into force mandating basic results

reporting of clinical trials.

Introduction

In the United States, the Food and Drug Administration (FDA) approves new drug products for sale and marketing based on results from clinical investigations that demonstrate the safety and efficacy of a drug for a proposed indication. Sponsors of a drug (e.g., companies, research institutions, or government) seek approval by submitting a new drug application (NDA) [1] to the FDA, which must include documentation and analyses of all animal and human trial data, as well as information about the ingredients, clinical pharmacology, manufacturing, processing, and packaging of the drug. The FDA relies on sponsors to submit all data, including complete protocols, protocol revisions, and data from failed trials in the NDA. The NDA is then reviewed by clinicians, statisticians, chemists, clinical pharmacologists, and other relevant scientific and regulatory disciplines within the FDA to confirm and validate the sponsor's conclusion that a drug is safe and effective.

For drugs that receive FDA approval, public disclosure of trial results may occur through a variety of sources. The FDA discloses a Summary Basis of Approval document that contains summaries and evaluations of clinical data and statistical analyses performed by FDA medical officers during the approval process [2]. However, these summaries contain only selected results from the clinical trials [1], and data deemed confidential or information considered commercial under Exemption 4 of the Freedom of Information Act may be redacted [3]. The drug label or package insert also provides a summary of clinical studies but often in less detail than the Summary Basis of Approval. Publication in the peer-reviewed medical literature is the main channel by which trial results are publicly disclosed and communicated to clinicians. The complete and accurate reporting of clinical trial results is crucial to ensuring an unbiased evidence base for advancing science and facilitating informed clinical decision-making [4], and has been considered an ethical obligation [5]. However, there was no requirement until very recently that trial results be published or otherwise made public for FDA-approved and marketed drugs.

A string of recent controversies concerning the suppression of safety risks of rosiglitazone [6], paroxetine [7], and rofecoxib [8,9] has drawn public attention to the limited and incomplete public access to clinical trial results on FDA-approved drugs [10] and has resulted in a concerted effort to achieve improved compliance with trial registration and greater disclosure of trial results [11–13].

In response to these concerns, the US recently mandated in the FDA Amendments Act 2007 (Public Law 110–85) that all trials supporting FDA-approved drugs and devices must be registered at inception and have their “basic results” publicly posted by the National Institutes of Health. The basic results to be disclosed include the demographics of the study participants, the number of participants who dropped out or were excluded from analysis, and the numeric and statistical test results of all primary and secondary outcomes declared at initial trial registration.

For the foreseeable future, however, the detailed information needed for full appraisal of a trial's evidence is likely to be available only in journal publications. This information includes protocol, protocol deviation, and conflicts of interest information, as well as additional analyses beyond the primary and secondary outcomes. The availability of basic results on ClinicalTrials.gov (http://www.clinicaltrials.gov/) will therefore complement, but not supplant, the medical literature's continuing role as the dominant channel of communication to clinicians and the public, even after the imposition of mandatory basic results reporting.

Previous research has documented the problem of publication bias and incomplete or selective reporting of trials submitted to licensing authorities in Sweden [14,15], Finland [14], and the US [10,16]. For example, among trials of antidepressants submitted to the FDA [16] or the Swedish drug regulatory authority [15], efficacy trials reporting positive results and larger effect sizes were more likely to be published. These analyses were limited to one drug class, specifically, antidepressants. Therefore, we evaluated the publication status of trials submitted to the FDA for a wide variety of approved drugs and identified factors associated with publication.

Methods

Identification of Clinical Trials

We identified all drugs approved by the FDA between January 1998 and December 2000 at the Center for Drug Evaluation and Research Web site, available at http://www.fda.gov/cder/da/da.htm. We included only new drugs classified as “new molecular entities,” which are drug products that have never been previously approved by the FDA for any indication, hereafter referred to as “new drug.” For each new drug, we retrieved the FDA Summary Basis for Approval and evaluated the medical and statistical review documents to identify clinical trials submitted by the sponsor. These review documents are available at http://www.accessdata.fda.gov/scripts/cder/drugsatfda/index.cfm for all new drugs approved since 1998.

Classification of Clinical Trials

Phase I trials are often small studies designed to provide supporting information about a drug's pharmacokinetic parameters, dosing schedule, common side effects, tolerability, and toxicity, but are limited by design or other factors in their ability to demonstrate efficacy. Phase II and III trials are often larger studies designed to provide evidence on the overall risks and benefits of a drug. The phase of a trial was often not reported in the FDA documents. Sponsors and the FDA frequently categorize certain trials as “pivotal.” These are trials that demonstrate the efficacy and safety of a drug for its proposed indication and provide the most useful information for clinical decision-making. Pivotal trials are typically Phase II or III trials, but there is no formal definition of a pivotal trial. In practice, trials that are reported in the “clinical studies” or “clinical efficacy” section of the FDA-approved drug label are considered pivotal. We used this scheme to categorize trials as “pivotal” or “nonpivotal.” We obtained the product label at the time of FDA approval for each new drug, or the next available product label if the initial product label was not available, at http://www.fda.gov/cder/approval/index.htm. Trials described in the summary documents for each drug approval that were also described in the “clinical studies” section of the corresponding drug label were categorized as pivotal. All other trials were categorized as nonpivotal.

Data Extraction

For each submitted trial, we recorded the following characteristics when available in the FDA documents: drug name (generic and trade), the number and location of study sites, the name of the principal investigator, the number of study participants, dosage and evaluation schedules, sample size, statistical significance of the primary outcome (p < 0.05 or confidence interval [CI] excluding no difference; or if the study was an equivalency study, p > 0.05 or CI including no difference or a CI excluding the prespecified difference described in the trial). Nonsignificant or null results are defined as p > 0.05 or CI including no difference; or if the study was an equivalency study, p < 0.05 or CI excluding no difference or a CI including the prespecified differences described in the trial. We also recorded whether the trial was randomized or double blinded as reported by the sponsor in the FDA documents.

Search Strategy and Publication Matching

We systematically searched common databases of biomedical journals that a typical clinician, consumer, or policy maker living in the US would reasonably search. These databases were PubMed, Cochrane Library, and the Cumulative Index for Nursing and Allied Health Literature (CINAHL). First, we electronically searched PubMed to match each trial identified in the FDA review documents to publications in the medical literature. We initially used the new drug's generic name and limited the search to Publication Type: Clinical Trial. All English-language retrievals were reviewed in abstract or full-text form. Trials identified in the FDA reviews were matched to a publication based on the following characteristics: drug name, sample size, dosing schedules, number and location of study centers, primary outcome measures, and statistical significance or estimated effect of the primary outcome results.

Only original research reports in full journal articles were counted as matching publications; abstracts or review articles were not considered matches, as these types of articles by definition contain incomplete descriptions of a trial's methods and results. For remaining trials that were not matched to a publication, we searched PubMed again without publication type limits using a variety of keywords (e.g., generic drug name; names of other drugs in the trial; disease/condition studied; outcomes measured; and trial characteristics such as “cross over”, “randomize”, “blind”, “washout”, “placebo”, “pharmacokinetics”, and “bioavailability”). If trials remained unmatched to a publication in PubMed after this more comprehensive search, we searched the entire Cochrane Library and CINAHL databases without limits using the similar keyword strategy described above. We also reviewed The Medical Letter (http://www.medletter.com/) for additional trial publications. For each trial, we verified statistical significance of primary results, randomization, double blinding, and sample size by reviewing the publication. We completed our literature search on 01 August 2006, which yields a follow-up period ranging between 5.5 to 8.5 y from the time of a new drug's approval (January 1998 to December 2000).

Statistical Analysis

The main outcome measures were time from FDA approval to publication of a full report, and whether a report was published by 2 or 5 y after approval. We analyzed publication at 2 y because pending Congressional legislation is considering mandating results reporting by 2 y after drug approval. Trials that were not published were censored as of 01 August 2006.

To control for multiple variables simultaneously, we carried out multivariate mixed effects logistic regression analysis and calculated odds ratios at 2 and 5 y after approval. All models were adjusted for clustering by drug (treated as a random effect). Predictors assessed in both univariate and multivariable analyses included statistical significance of the primary results, double blinding, randomization, sample size (dichotomized at the median size of ≤ 135 or > 135, or log-transformed to each 2-fold increase in sample size), study type (pivotal or not), and company size. Companies with annual revenues greater than $3 billion, and/or annual research and development expenditures greater than $500 million in 2004, were classified as large companies, and generally represented the top 30 pharmaceutical and biotechnology companies in the world [17]. Month zero was defined as the month of FDA approval as stated in the FDA documents. The publication month was the month of the journal issue in which the trial appeared. Trials published before their FDA approval date were analyzed as published at time zero. In cases of duplicate publication (those reporting the same findings and results from the same trial, study population, intervention, and measured outcomes), we included only the earliest publication in all analyses. We chose variables for inclusion in multivariable models using forward stepwise selection with p < 0.05 required for entry and retention.

Our primary analysis was logistic regression analyses on all supporting trials (n = 909 trials). Our secondary analysis was on the subset of trials classified as pivotal (n = 340 trials). Data were analyzed with SAS software (version 9.1, SAS Institute).

Results

All Supporting Trials

We identified 90 FDA-approved new drugs between January 1998 and December 2000. Eighty-nine (99%) of the applications were submitted by a pharmaceutical company; one application was submitted by the US Army Medical Research and Material Command. Eighty-eight drugs were available by prescription only and two had over-the-counter marketing status. Seven prescription drug products were discontinued after initial FDA approval. We were able to identify a total of 909 trials with sufficient description in the FDA review documents supporting these 90 new drugs. Table 1 describes the trials' characteristics. We matched 394 of these trials (43%) to publications in the medical literature (Figure 1): 393 to publications in PubMed, the Cochrane Library, or the CINAHL database, and one to a publication cited by The Medical Letter but not indexed by the searched databases. The remaining 515 trials (57%) could not be matched to any publication. The proportion of trials published per new drug ranged from 0% to 100%, with an average of 55% of supporting trials published per new drug (Table S1). One of the 90 new drugs, an antibiotic, had none of its supporting trials published. Duplicate publications were seen in six trials: five trials had results published twice and one trial had results published three times, to total 401 matched publications.

In univariate analyses of all supporting trials, trials with statistically significant results, larger sample sizes, double blinding, randomization, and trials that were pivotal were more likely to be published by 2 and 5 y after FDA approval (Table 2). Company size did not appear to be associated with publication. When controlling for all of these factors simultaneously in multivariable analyses, statistically significant results, larger sample sizes, and pivotal status continued to be strong predictors of publication at 2 and 5 y after FDA approval (Table 3). Adding an interaction of statistically significant results and sample size estimated the effect of sample size to be smaller for studies with statistically significant results by a factor of 0.81 (p = 0.19) at 2 y and 0.79 (p = 0.14) at 5 y. Because statistical significance was missing for many studies, we also fit models like those in Tables 2 and 3, but with “unknown” statistical significance counted as a third possible category. This permitted inclusion of 883 trials, but produced no qualitative changes in the results. Trials with unknown statistical significance were estimated to be less likely to be published than trials with nonsignificant results at 2 y (OR 0.71, p = 0.28) and 5 y (OR 0.59, p = 0.067) with a nearly unchanged estimate of the effect of statistical significance (OR 2.53, p = 0.001 at 2 y, OR 3.06, p < 0.001 at 5 y). Results from Cox proportional hazards modeling with a shared gamma frailty were qualitatively similar to the random effects logistic regression results and so are not shown.

Figure 2 shows the yearly number and cumulative proportion of trials published relative to the time of FDA approval. Thirty-two percent (128/394) of the publications occurred prior to the relevant new drug's FDA approval and 92% (364/394) were published within 3 y of FDA approval. Among published trials reporting the statistical significance of their primary outcome (n = 337), the median time to publication from FDA approval for trials with statistically significant results was 0.77 y (range 0–4.41 y, n = 285) and 0.73 y for trials without statistically significant results (range 0–3.84 y, n = 52).

Pivotal Trials

Of the 909 trials, 340 (37%) were identified as pivotal, of which 257 (76%) were published (Figure 1). The predictors of publication for pivotal trials were similar to those for all supporting trials in univariate (Table 2) and multivariate analyses (Table 3). Interaction of statistically significant results and sample size was similar to that for all trials, with the effect of sample size estimated to be smaller for studies with statistically significant results by a factor of 0.82 (p = 0.54) at 2 y and 0.83 (p = 0.54) at 5 y. Like the analysis of all trials, counting unknown statistical significance as a valid third category permitted inclusion of more trials (n = 339) but produced no qualitative changes in results. Figure 3 shows the yearly number and cumulative proportion of pivotal trials published relative to the time of FDA approval. Thirty-two percent (82/257) of the publications occurred prior to the relevant drug's FDA approval and 95% (245/257) were published within 3 y after FDA approval.

Discussion

Our study evaluated the publication of 909 clinical trials identified in FDA medical and statistical review documents in support of 90 new drug products approved between 1998 and 2000. We found that after a minimum of 5.5 y of follow-up after FDA approval, we identified publications from 43% of the trials in the medical literature. For pivotal trials, which are more clinically informative than nonpivotal trials, we found publications from 76% of the trials. For one of the 90 approved new drugs, we could not find any published supporting trial. We also found strong evidence of publication bias: trials with statistically significant results were more likely to be published than trials with nonsignificant results, as were trials with larger sample sizes. There was a weak suggestion that the effect of sample size might be less among trials with statistically significant findings, but p-values for such interactions did not reach statistical significance. Our study therefore shows that previous findings of publication bias of trials supporting the regulatory applications of selected drug classes (e.g., antidepressants) [10,14–16] are broadly true across a diverse group of drug classes. Publication bias may lead to an inappropriately favorable record in the medical literature of a drug's true risk/benefit profile relative to other standard therapies, and may thus lead to preferential prescribing of newer and more-expensive treatments. We could not test whether similar publication bias exists for trials supporting unsuccessful new drug applications because adequate information about these applications was unavailable from the FDA or other government or commercial sources.

We also found the reporting of clinical trials in the FDA review documents and drug labels to be variable in detail and content, and not an adequate substitute for full publication in the medical literature. For example, reporting ranged from detailed descriptions of a trial's study design, intervention, patient population, statistical analyses, adverse events, primary outcomes, and other results, to brief statements that only summarized a trial's primary outcome. We also noted sections of redacted information in the FDA review documents. Neither the FDA review documents nor the drug labels followed a standard format for reporting a trial's methodology and results. Use of guidelines such as the revised CONSORT (Consolidated Standards of Reporting Trials) [18] may help to improve the quality and completeness of trial reporting in FDA review documents as others have proposed [19].

Our study has several limitations. First, we may have misclassified some published trials as being unpublished because of difficulties in matching publications to incomplete trial descriptions in the FDA documents. Also, we did not search other databases such as the European EMBASE, nor did we contact investigators or sponsors to determine publication status or verify that a trial was not published or in press. Thus, we are likely to have underestimated the overall publication rate of these trials. However, we believe that for clinicians and policy makers, the most relevant publication rate is not the overall rate but the publication rate in journals that a typical clinician, consumer, or policy maker would have access to through a reasonable literature search. We believe our searches of PubMed, the Cochrane Library, and CINAHL reflect such a reasonable search. It would not be reasonable to expect a clinician, consumer, or policy maker to contact investigators or sponsors to determine a trial's publication status.

A second limitation of our study is our follow-up time of 5.5 to 8.5 y after new drug approval may be inadequate. However, we found that publications occurred almost exclusively within the first 3 y after approval, making it unlikely that longer follow-up would yield many additional publications. Third, time-to-publication is ideally counted from the date of trial completion, but we were unable to obtain these dates reliably. Moreover, we believe the month of approval is the most relevant time point when trial results should be available to the public. Fourth, our study focused on publications in the medical literature, but some companies have started making their trial results publicly available directly on their own Web sites. For example, the pharmaceutical industry's Clinical Study Results Database contains summaries of “hypothesis-testing” trials completed since October 2002 for many pharmaceutical products [20]. We searched this database for the 515 unpublished trials and found summaries for 22 (4%) of them. The effect of this and other related Web sites on public disclosure of trial data submitted to the FDA requires further research as the information reported in these databases may not be peer reviewed and there is no guarantee that the reporting is complete for all relevant data. Fifth, we could not determine the statistical significance of the findings of a substantial proportion of the studies. We did, however, obtain qualitatively similar results when we performed a sensitivity analysis by counting unknown statistical significance as a valid third category. Finally, our findings cannot be generalized to any specific product, company, institution, organization, or investigator.

Despite these limitations, our study provides ample evidence that in the years immediately following FDA approval that are most relevant to public health, there exists incomplete and selective publication of trials supporting approved new drugs. Potential reasons for this publication bias may include the tendency of investigators and sponsors to delay or not submit trial reports [21,22], or the motivation of commercial sponsors to publish positive trials in prestigious journals to obtain article reprints for marketing [23]. Bias in editorial decisions toward publishing positive results is also possible, although there is evidence suggesting that this is not the case [24,25]. Regardless of the cause, publication bias harms the public good by impairing the ability of clinicians and patients to make informed clinical decisions, and the ability of scientists to design safer and more efficient trials based on past findings. Publication bias can thus be considered a form of scientific misconduct [5].

Potential Effects of Mandatory Results Reporting on Publication Bias

As discussed above, the FDA Amendments Act of 2007 mandates basic public results reporting for all trials supporting FDA-approved drugs and devices. Our study shows that this legislation was necessary because current reporting is marked by pervasive publication bias of positive over negative trials. Moreover, because published trial reports are often incomplete [26] and have been shown to selectively report favorable outcome results [27], the published evidence supporting FDA-approved drugs may be even more skewed than our results suggest. By ensuring the reporting of all predeclared primary and secondary outcomes regardless of their direction of benefit, the new law should go a long way toward correcting this skew.

We anticipate that the new law will also speed the dissemination of trial information. Currently, according to our data, 40% of the trials that were eventually published were published more than 1 y postapproval (34% of pivotal trials). Under the new law, basic results for all trials must be posted by 1 y after trial completion or approval of the drug or device. This suggests that for all trials that the sponsor wishes to publish, the manuscripts will have to be submitted for peer review before the 1 y postapproval mark if they hope to allay journal concerns about publishing trials whose primary and secondary outcome results have already been publicly posted. Thus, we would expect the time-to-publication curves in Figures 2 and 3 to shift left.

Paradoxically, however, this new law may increase rather than decrease publication bias. Might sponsors feel less compelled to publish equivocal trials because the basic results will already be in the public domain? Might the time pressure to submit manuscripts by 1 y postapproval focus sponsor efforts even more on submitting positive trials and trials of greatest interest to journals? Might the journals, if they accept manuscripts of trials with publicly posted results, change the criteria by which publication importance is judged, and how might this affect acceptance rates [28]? When more detailed protocol information must also be posted on ClinicalTrials.gov, to start no later than October 2010, the effect on publication practices is even harder to anticipate. Our data document the current degree of publication bias and provide a baseline for assessing the evolving publication practices of trials supporting FDA-approved drugs as mandatory basic results reporting takes effect.

Supporting Information

Attachment 1