Completeness of Reporting of Patient-Relevant Clinical Trial Outcomes: Comparison of Unpublished Clinical Study Reports with Publicly Available Data

Summary

Background:
Access to unpublished clinical study reports (CSRs) is currently being discussed as a means to allow unbiased evaluation of clinical research. The Institute for Quality and Efficiency in Health Care (IQWiG) routinely requests CSRs from manufacturers for its drug assessments.

Our objective was to determine the information gain from CSRs compared to publicly available sources (journal publications and registry reports) for patient-relevant outcomes included in IQWiG health technology assessments (HTAs) of drugs.

Methods and Findings:
We used a sample of 101 trials with full CSRs received for 16 HTAs of drugs completed by IQWiG between 15 January 2006 and 14 February 2011, and analyzed the CSRs and the publicly available sources of these trials. For each document type we assessed the completeness of information on all patient-relevant outcomes included in the HTAs (benefit outcomes, e.g., mortality, symptoms, and health-related quality of life; harm outcomes, e.g., adverse events). We dichotomized the outcomes as “completely reported” or “incompletely reported.” For each document type, we calculated the proportion of outcomes with complete information per outcome category and overall.

We analyzed 101 trials with CSRs; 86 had at least one publicly available source, 65 at least one journal publication, and 50 a registry report. The trials included 1,080 patient-relevant outcomes. The CSRs provided complete information on a considerably higher proportion of outcomes (86%) than the combined publicly available sources (39%). With the exception of health-related quality of life (57%), CSRs provided complete information on 78% to 100% of the various benefit outcomes (combined publicly available sources: 20% to 53%). CSRs also provided considerably more information on harms. The differences in completeness of information for patient-relevant outcomes between CSRs and journal publications or registry reports (or a combination of both) were statistically significant for all types of outcomes.

The main limitation of our study is that our sample is not representative because only CSRs provided voluntarily by pharmaceutical companies upon request could be assessed. In addition, the sample covered only a limited number of therapeutic areas and was restricted to randomized controlled trials investigating drugs.

Conclusions:
In contrast to CSRs, publicly available sources provide insufficient information on patient-relevant outcomes of clinical trials. CSRs should therefore be made publicly available.

Please see later in the article for the Editors' Summary

Introduction

Publication bias and outcome reporting bias pose a substantial threat to the validity of clinical research findings and thus to informed decision-making in health care [1,2]. In recent years major initiatives to prevent or at least identify these biases have been implemented, such as registration of clinical trials as a precondition for publication in medical journals in 2005 [3], or mandatory trial registration and reporting of methods and results in ClinicalTrials.gov following the Food and Drug Administration Amendments Act of 2007 [4]. However, the application of these measures has been insufficient [5–8], and they also contain several loopholes [9]. For instance, the measures do not apply to clinical trials completed before 2005 and 2007, respectively, and provide only summarized information, preventing full evaluation.

Various types of formats exist for reporting clinical trials of drugs: journal publications and reports from trial registries and results databases—hereafter referred to as “registry reports”—make summaries of trials publicly available (e.g., to clinicians and authors of systematic reviews). These publicly available formats currently represent the main information source for clinical and health policy decision-making. Reporting standards for these two formats include the Consolidated Standards of Reporting Trials (CONSORT [10]) for journal publications and the Food and Drug Administration Amendments Act for registry reports on trials of US Food and Drug Administration–regulated drugs and medical devices [4]. In contrast to the first two formats, clinical study reports (CSRs) are detailed accounts of trials generally prepared following the International Conference on Harmonisation's Guideline for Industry: Structure and Content of Clinical Study Reports (ICH E3 [11]). The value of additional information from CSRs in drug assessment has been shown in the cases of the antiviral oseltamivir (Tamiflu) and the antidepressant reboxetine, in which conclusions on these drugs based on published evidence alone were challenged and in part even reversed by unpublished information from CSRs [12,13].

So far, CSRs are used to inform regulatory decision-making, but are in general not publicly available. The few cases in which CSRs have been used for drug evaluation outside regulatory agencies required major efforts by researchers to gain access to the documents [14–16]. However, the European Medicines Agency (EMA) has launched an initiative to improve transparency in clinical research by providing unpublished clinical trial data [17,18]. This initiative also involves a discussion of the data formats to be made publicly available [19], and CSRs are being considered, in addition to individual patient data [20]. Furthermore, legal measures to improve transparency have been proposed by the European Commission and the European Parliament [21,22], also addressing the extent of trial data to be published. Thus, the role of CSRs for the evaluation of clinical trials is currently of particular importance, and we would like to further inform the current debate with our experiences.

Health Technology Assessments of Drugs at the Institute for Quality and Efficiency in Health Care

The Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; IQWiG), established in 2004, is Germany's main health technology assessment (HTA) agency. Its primary responsibility is the production of HTA reports on drugs and non-drug interventions based on the analysis of patient-relevant outcomes, i.e., outcomes describing morbidity, mortality, and health-related quality of life (HRQoL). These reports inform health policy decision-making in the German statutory health care system. IQWiG attempts to obtain the most complete information possible for its HTAs. For this purpose, during the preparation of a drug report, besides systematically searching bibliographic databases and trial (results) registries, we routinely ask the manufacturer to provide an overview of sponsored published and unpublished clinical trials of the drug under assessment. From this list we select the trials deemed relevant to the assessment and ask the manufacturer to submit the full CSRs. However, except for early assessments of new drugs (which are not the subject of this article), the manufacturer is not obliged to provide CSRs.

Previous Study of Clinical Study Reports versus Publicly Available Sources

In a previous study investigating the availability of information on methods and selected outcomes of clinical trials in different types of reporting formats, we used the pool of randomized controlled trials and corresponding documents (CSRs, journal publications, registry reports) included in HTAs of drugs prepared by IQWiG (see below). This previous study showed that journal publications and registry reports had different strengths and weaknesses and that, overall, the CSRs provided considerably more complete information on items relating to methods and selected outcomes than publicly available sources [23].

Rationale for Current Study

The previous study investigated only a limited range of outcomes, i.e., primary outcomes (irrespective of whether they were patient-relevant or not) and some adverse event (AE) outcomes. However, as stated, our HTAs are generally based on a wide range of patient-relevant outcomes (irrespective of whether they are primary outcomes or not). We hypothesized that the information gain from CSRs versus publicly available sources could be even greater for patient-relevant outcomes (which are often non-primary) than for the subset of outcomes investigated in the previous study. In the current study we therefore investigated the information gain for all patient-relevant outcomes included in our HTAs. We also aimed to characterize the information gain from CSRs for various types of patient-relevant outcomes.

Methods

The methods for the current study were largely based on those described previously [23]. In the previous study, we included all HTAs of drugs finalized by IQWiG between 15 January 2006 and 14 February 2011, which—besides a systematic search for journal publications—contained a systematic search for registry reports as part of the information retrieval process. The systematic search generally covered MEDLINE, Embase, and the databases of the Cochrane Library, as well as ClinicalTrials.gov, the International Clinical Trials Registry Platform of the World Health Organization, the Clinical Trials Portal of the International Federation of Pharmaceutical Manufacturers and Associations, the Clinical Study Results Database of the Pharmaceutical Research and Manufacturers of America, and the trial registries and results databases of the manufacturers of the drugs under investigation. In addition, for all HTAs, CSRs were requested from the manufacturers of the drugs under assessment.

In the previous study we included all 286 trials and corresponding documents (101 CSRs, 192 journal publications, and 78 registry reports) considered in the 16 HTAs. For the current study, we used the same pool of HTAs, but included only the 101 trials from the original pool of 286 trials for which the manufacturer had provided a full CSR. “Full” referred to the availability of a core text (including a full description of methods and results) and all tables and figures, as well as appendices (e.g., protocol or statistical analysis plan) if they were referenced in the core text with only insufficient information provided in the text. None of these CSRs were publicly available at the time of preparation of the HTAs.

As stated, the previous study investigated the reporting of only a limited set of trial outcomes in the various reporting formats. In contrast, our current study aimed to characterize reporting in CSRs versus publicly available documents for all patient-relevant outcomes considered in our HTAs. These outcomes had been prespecified in the HTA protocols during the preparation of the 16 HTAs, and had been identified in the three reporting formats by systematically screening all of the available CSRs, registry reports, and journal publications.

We entered all data for the previous and current study into a Microsoft Access database. The database contained information on the characteristics of the HTA, the type of document available, as well as basic trial and document characteristics (see Table 1). For the current study we also entered data on the reporting quality of all patient-relevant outcomes as described below. In addition, we classified these outcomes as mortality, clinical events, symptoms, or HRQoL (benefit outcomes), as well as AEs, serious AEs (SAEs), AEs of special interest, or withdrawals due to AEs (harm outcomes); please see Table 2 and Box 1 for coding definitions.

Our requirements for complete reporting of patient-relevant outcomes were based on the requirements of authors of systematic reviews (i.e., provision of adequate information for assessment of risk of bias and adequate data for meta-analyses) [24]. Completeness of the information provided for the patient-relevant outcomes was recorded as (1) completely reported including numerical data, (2) partly reported including numerical data, (3) verbally reported without numerical data, or (4) not reported. A definition of all categories is provided in Table 2. In the assessment of completeness of information in journal publications, if more than one journal publication was available and an outcome was completely reported in one publication but not in the other(s), reporting of the outcome was still classified as “complete.”

All data for the current study were extracted and coded by one author. All data from registry reports and all classifications of patient-relevant outcomes were independently checked by a second author. In addition, a random sample of 10% of the data and codings for trial outcomes from CSRs and journal publications was also independently checked by a second author (agreement between authors for CSRs: 99%; for journal publications: 97%). Discrepancies were resolved by consensus, if necessary, after discussion with a third author.

To quantify the information gain through CSRs, we calculated the proportion of outcomes with complete reporting (category 1 above) and incomplete reporting (categories 2–4 above) for CSRs and publicly available sources (journal publications, registry reports, and the combination of both). Besides presenting the dichotomous categories “complete reporting” versus “incomplete reporting,” we also presented separately the three categories of incomplete reporting (categories 2–4 above). In addition, we performed direct comparisons of trials for which CSRs as well as journal publications and/or registry reports were available. To investigate completeness of reporting over time, we calculated the proportion of outcomes with complete reporting in the different document types stratified by year of finalization of the CSRs.

The proportion of outcomes with complete reporting was compared between CSRs and journal publications or registry reports (or a combination of both) using the McNemar test to take the potential dependency of samples into account. The data were analyzed using SAS 9.2.

The manuscripts of the previous and current study show a minor overlap of results data: as AEs were investigated under the research questions of both studies, both manuscripts report the proportion of outcomes with complete information in CSRs for AEs (92%), SAEs (88%), and withdrawals due to AEs (91%) (see Table 3).

Results

Table 1 shows the characteristics of the trials, documents, and patient-relevant outcomes included in our sample. We analyzed 101 trials with CSRs. These CSRs were prepared between 24 September 1989 and 29 January 2010. The pool of clinical trials included nearly 70,000 patients and covered six different therapeutic areas (mainly depression and type I and II diabetes; the drugs assessed are listed by therapeutic area in Table 4). Of the 101 trials, 90 were efficacy trials; 86 had at least one publicly available source, 65 had at least one journal publication, and 50 had a registry report. For 15 trials, the CSR was the only source of information available.

The 101 trials included 1,080 outcomes classified by IQWiG as patient-relevant and considered in the pool of HTAs. Among the benefit outcomes, symptoms were investigated most often, whereas HRQoL was investigated least often. Among the harm outcomes, overall rates of AEs, SAEs, and withdrawals due to AEs were available for each trial; the harm outcomes considered most often were AEs of special interest in the given indication.

Overall Completeness of Information in Clinical Study Reports versus Publicly Available Sources

Table 3 shows the completeness of information for trial outcomes by reporting format in the full trial sample. The CSRs provided complete information on a considerably higher proportion of patient-relevant outcomes (86%) than journal publications and registry reports, even if these two sources were combined (39%). With the exception of HRQoL (57%), CSRs provided complete information on 78% to 100% of benefit outcomes. The highest value was achieved for mortality (100%). The corresponding values for combined publicly available sources were considerably lower; completeness of reporting ranged from 20% to 53%. CSRs provided complete information on 84% to 92% of harm outcomes. Again, the corresponding values for the combined publicly available sources were considerably lower (27% to 72%). The comparison of journal publications and registry reports showed that, overall, completeness of information was similar for benefit outcomes (19%) and harm outcomes (25% to 26%). However, when specific outcomes were considered, the two reporting formats showed different levels of completeness (e.g., for clinical events or the overall rate of AEs).

The differences in completeness of information for patient-relevant outcomes between CSRs and journal publications or registry reports (or a combination of both) were statistically significant for all types of outcomes (see Table 5).

Publication Bias and Outcome Reporting Bias

In addition to analyzing the proportion of outcomes for which complete information was available, we also aimed to further describe the reporting of outcomes with incomplete information. Table 6 presents the pattern of reporting of all patient-relevant outcomes in journal publications and/or registry reports. The data show that most outcomes that were not reported completely were not reported at all, except for outcomes on symptoms and HRQoL, which were reported partly with data or only verbally without data in 35% to 40% of cases. However, also for these two outcomes a large proportion of outcomes were not available at all from publicly available sources. Non-availability of outcomes was due either to lack of reporting of these outcomes even though a publication and/or registry report was available (34% of all outcomes, outcome reporting bias) or to lack of reporting of the whole trial (13%, publication bias). Tables S1 and S2 show the same type of analysis for journal publications and registry reports separately. Table 4 provides examples of the numerous patient-relevant outcomes (including outcomes of major clinical relevance, such as overall mortality and potentially life-threatening events) not reported in the publicly available sources, by therapeutic area and outcome category.

Matched Pairs of Clinical Study Reports versus Publications and/or Registry Reports

The results presented so far describe the completeness of information in publicly available reporting formats for a given sample of trials (101 trials with CSRs). Part of the differences described resulted from the fact that journal publications or registry reports were not available for all trials in our sample, as it also included unpublished trials only reported in CSRs. To investigate whether CSRs provided superior information when they were directly compared to the corresponding journal publications and registry reports, we analyzed the completeness of information in CSRs versus the publicly available sources in samples including only trials for which the respective source was available (see Tables 7 and S3–S5). Overall, each of these analyses confirmed that a substantial amount of additional information on patient-relevant outcomes is gained from CSRs compared to journal publications or registry reports (or a combination of both), even for published trials.

Completeness of Reporting over Time

To investigate completeness of reporting over time, we analyzed the availability of trial reports in publicly available sources as well as the proportion of completely reported outcomes in the three document types over time (Table 8). The analysis showed an increasing availability of trials in combined publicly available sources over time (from 71% to 95%). However, the proportion of trials available in journal publications dropped to about 50% for trials with CSRs finalized between 2005 and 2010. We hypothesized that this decrease could have been caused by the fact that the temporal proximity of the literature searches in the HTAs and the finalization date of the CSR had not allowed sufficient time for preparation and publication of a manuscript. We therefore performed a sensitivity analysis in which all trials with a CSR finalization date less than 2 y before the search date of the HTA were classified as published in a journal. This “best-case scenario” resulted in an availability rate of trials in journal publications of 81%. We also performed the same type of analysis for registry reports; the corresponding rate was 93%.

However, in our sample, high availability rates of trial reports in publicly available sources did not result in high rates of completely reported patient-relevant outcomes: for instance, even for trials for which the availability rate in combined publicly available sources was more than 90%, less than 50% of patient-relevant outcomes were completely reported. In contrast, after 1995, CSRs consistently provided complete information for more than 90% of patient-relevant outcomes.

Discussion

Summary of Findings

To our knowledge the current study quantifies for the first time how much information on a wide range of patient-relevant outcomes included in a large pool of clinical trials can be gained from making full CSRs available. Our findings show that a substantial amount of information on patient-relevant outcomes required for unbiased trial evaluation is missing from the public record. This is all the more important as such outcomes are preferably considered in comparative effectiveness research and consequently in health policy and clinical decision-making [25,26]. At the same time, this information can be obtained from CSRs, i.e., from documents routinely prepared by sponsors of clinical trials, but not usually made publicly available. Over twice as much information on patient-relevant outcomes can be gained from CSRs than from publicly available sources (86% versus 39% completely reported outcomes). Moreover, CSRs not only provide patient-relevant information in cases where journal publications and registry reports are missing, they also present additional information in cases where trials have been reported in journals or registries. The differences in information gain from the different reporting formats are due to a general superiority of CSRs over publicly available sources, demonstrated by the higher proportion of completely reported outcomes in a matched sample of CSRs and publicly available documents. Our findings also again confirm the existence of considerable publication and outcome reporting bias in clinical research: 36% of the trials in our pool were not published in journal publications, 15% had no publicly available reports at all, and even for trials with publicly available reports, 34% of patient-relevant outcomes, including outcomes of major clinical relevance, were not reported.

Our analysis of completeness of reporting over time showed that although the rate of trials made available in journal publications and registry reports is increasing, the rate of completeness of information on patient-relevant outcomes in these sources is not. These findings show that new approaches are needed. It is insufficient to aim for a journal publication rate of 100%. What is needed is public availability of CSRs, and thus of documents presenting trial results to a level of detail required for full evaluation of a trial.

Comparison with Previous Research

Because of the fact that CSRs are generally not publicly available, only a few researchers have investigated their content as well as their possible role in providing information on clinical trials. Doshi and Jefferson analyzed a sample of 78 CSRs and showed that CSRs had a median length of about 450 pages of text and main tables plus an additional 550 pages of efficacy and safety listings [27]. Vedula et al. compared unpublished internal company documents from the gabapentin litigation case (unpublished protocols, statistical analysis plans, and research reports) with trial publications [28,29]. Besides identifying several inconsistencies in the corresponding trial publications, they found that the unpublished documents provided more extensive documentation of methods planned and used, as well as trial findings. The research already cited analyzing CSRs on oseltamivir (Tamiflu) and reboxetine showed that prior conclusions on a drug's benefits and harms based on published evidence alone could no longer be upheld when information from CSRs became available [12,13]. Our previous study of CSRs showed that considerably more relevant information on trial methods, primary outcomes, and some AE outcomes can be gained from CSRs [23]. In the current study, information gain from CSRs versus publicly available sources was even higher for a full set of patient-relevant outcomes than for the limited set of trial outcomes investigated in our previous study. While the proportion of completely reported primary and AE outcomes in the previous study was 91% for CSRs, 52% for journal publications, and 71% for registry reports [23], the corresponding values for the full range of (primary and non-primary) patient-relevant outcomes investigated in the current study were 86%, 23%, and 22%, respectively.

Relevance of Full Trial Information for Everyday Patient Care

Our findings suggest that oseltamivir and reboxetine might not be the only cases in which conclusions on benefits and harms might be changed by making full information on all clinical trials available to independent researchers and subsequently to clinicians and patients. Access to CSRs would thus allow informed decision-making and directly influence patient care.

The goal of assessing the full information from CSRs is not only to determine the benefits and harms of a single drug, but also to investigate the position of a drug in the given therapeutic area. For this purpose, comparative effectiveness research is gaining momentum both in the US and in Europe [25,30]. This area of research would specifically benefit from full CSRs being publicly available. As direct comparisons of alternative treatment methods are not available for all comparative effectiveness research questions, indirect comparisons will become more important, and CSRs are essential sources to inform meaningful indirect comparisons. This is because, firstly, indirect comparisons require detailed information on methods (i.e., a full protocol) of the clinical trials of interest, as well as on the trial population, to assess whether indirect comparisons within a given pool of trials are appropriate at all; this type of information is available in CSRs. Secondly, indirect comparisons require full numerical information on all relevant outcomes for network meta-analyses; as our analyses show, such extensive information is provided only in CSRs.

How Can Full Access to Clinical Study Reports Be Achieved?

As stated, the EMA intends to proactively publish complete clinical trial data, possibly including CSRs, from January 2014 onwards [18], and for this purpose has held extensive consultations with advisory groups of stakeholders and other interested parties [19] and has published a draft policy [20]. An even clearer solution would be a legal requirement to make CSRs publicly available, as is currently being discussed for the planned European legislation on clinical research [22].

However, both initiatives have a potential major flaw: they probably would apply only to drugs approved from January 2014 onwards, or for trials conducted after new legislation came into effect. This would present a problem because most drugs in current use would not be covered by the new measures, yet these drugs will still be widely used in clinical practice for years to come. Thus, although comprehensive information would in future be available for newer drugs, published information on the majority of drugs would still remain biased. This would hamper a meaningful comparison of alternative treatment methods. In addition, open questions about drugs in current use may never be answered. This is particularly relevant for drugs with a substantial public health impact, such as oseltamivir. The CSRs in our pool of trials were prepared between 1985 and 2010 and prove the value of CSRs for drugs in current use. The CSRs of such drugs that were submitted to regulatory authorities should therefore be made publicly available in a central repository to complete the evidence base. Pharmaceutical companies and non-industry trial sponsors could also release CSRs, thus underlining their commitment to transparency.

In line with our point of view, a further initiative to promote trial registration and reporting of full methods and results, the AllTrials initiative (http://www.alltrials.net/), also specifically refers to “past and present” clinical trials.

Further Rocks on the Road to Full Data Transparency

It should be noted that the full implementation of the new EMA policy is in jeopardy as the pharmaceutical industry, which has previously expressed its reservations about the policy [31], is taking legal action: two Freedom of Information requests were made to the EMA under its current data transparency policy to release individual patient data for adalimumab (Humira), a tumor necrosis factor inhibitor approved for rheumatoid arthritis and other indications. However, the company AbbVie has sought an injunction to block the EMA from releasing the data. A second company, Intermune, has also taken legal action against the EMA [32]. The interim decision by the General Court of the European Union is in favor of the companies: the EMA has been ordered not to provide documents until a final ruling is given by the Court [33]. The two court cases seem to represent not just the policy of single companies but a general industry strategy, since both European and US pharmaceutical industry bodies have lodged supportive pleas [32]. This action contradicts repeated assertions by industry that it supports data transparency.

Limitations

Our study has a number of limitations. First of all, we were not able to investigate a representative or random sample of CSRs, because these documents are usually not available outside pharmaceutical companies and regulatory agencies. Therefore, our sample was based on CSRs provided voluntarily by pharmaceutical companies upon request during our assessment procedures. We did not receive a (full) CSR for 62% (167/268) of the trials included in our HTAs and thus had to exclude these trials from our current study. The excluded and included trials showed differences in the therapeutic areas investigated (see Table S6), for example, the former comprised a higher proportion of trials on depression (57% versus 40%), but a lower proportion of trials on diabetes (21% versus 44%). In addition, a higher proportion of the excluded trials were reported in journal publications (76% versus 65%), whereas a lower proportion of these trials were reported in registry reports (17% versus 50%). It is unclear whether our results would have been different if they had been based on a random sample of CSRs. Furthermore, our sample covered only a limited number of therapeutic areas and was restricted to randomized controlled trials investigating drugs, so we cannot comment on other trial designs or trials of non-drug interventions. In addition, the registry reports included were generated by a limited number of pharmaceutical companies and were not prepared according to the requirements of the Food and Drug Administration Amendments Act [23]; future reports in ClinicalTrials.gov may be of better quality.

The dataset for our study was generated in 2011. We did not perform an update of the dataset as this would have required a major investment of resources. However, we believe that this dataset, which includes a total of 101 trials with 1,080 patient-relevant outcomes, is large enough to produce meaningful results.

We also note that several further issues related to CSRs could be investigated in future research. For example, except for the case of reboxetine [13], we can make statements only about the completeness of information in CSRs versus publicly available sources; we did not analyze how often the inclusion of data from CSRs changed the interpretation of the overall results of a study. Moreover, we did not investigate whether certain study characteristics (e.g., enrollment size) influenced completeness of reporting, nor did we specifically describe study protocols included in CSRs.

Conclusion

Information on patient-relevant outcomes investigated in clinical trials is insufficient in publicly available sources; considerably more information can be gained from CSRs. CSRs should be made publicly available as they may substantially influence conclusions concerning the actual position of an individual drug in a therapeutic area. Our findings underline the importance of CSRs—both for past and future trials—for unbiased trial evaluation, thus supporting informed decision-making in health care.

Supporting Information

Attachment 1

Attachment 2

Attachment 3

Attachment 4

Attachment 5

Attachment 6