Differences in Reporting of Analyses in Internal Company Documents Versus Published Trial Reports: Comparisons in Industry-Sponsored Trials in Off-Label Uses of Gabapentin

Summary

Background:
Details about the type of analysis (e.g., intent to treat [ITT]) and definitions (i.e., criteria for including participants in the analysis) are necessary for interpreting a clinical trial's findings. Our objective was to compare the description of types of analyses and criteria for including participants in the publication (i.e., what was reported) with descriptions in the corresponding internal company documents (i.e., what was planned and what was done). Trials were for off-label uses of gabapentin sponsored by Pfizer and Parke-Davis, and documents were obtained through litigation.

Methods and Findings:
For each trial, we compared internal company documents (protocols, statistical analysis plans, and research reports, all unpublished), with publications. One author extracted data and another verified, with a third person verifying discordant items and a sample of the rest. Extracted data included the number of participants randomized and analyzed for efficacy, and types of analyses for efficacy and safety and their definitions (i.e., criteria for including participants in each type of analysis). We identified 21 trials, 11 of which were published randomized controlled trials, and that provided the documents needed for planned comparisons. For three trials, there was disagreement on the number of randomized participants between the research report and publication. Seven types of efficacy analyses were described in the protocols, statistical analysis plans, and publications, including ITT and six others. The protocol or publication described ITT using six different definitions, resulting in frequent disagreements between the two documents (i.e., different numbers of participants were included in the analyses).

Conclusions:
Descriptions of analyses conducted did not agree between internal company documents and what was publicly reported. Internal company documents provide extensive documentation of methods planned and used, and trial findings, and should be publicly accessible. Reporting standards for randomized controlled trials should recommend transparent descriptions and definitions of analyses performed and which study participants are excluded.



Please see later in the article for the Editors' Summary

Introduction

Intent to treat (ITT) analysis, which includes all participants in the groups to which they were randomized, is typically recommended as the primary type of analysis for randomized controlled trials (RCTs) [1]–[3]. An ITT analysis preserves the random assignment of participants to the study groups and thereby protects against selection bias while estimating the treatment effect, when no outcome data are missing. In addition to ITT, other types of analysis (e.g., per protocol analysis) are often conducted in RCTs. Such additional analyses, which include only a subset of all participants in the trial, can be used to assess sensitivity of findings compared to the primary analysis. Because each type of analysis estimates a different parameter (i.e., population effect) and exclusion of participants from an analysis can potentially bias its estimate (i.e., observed effect), details on the types of analysis and their definitions (i.e., criteria for including participants in the analysis), and the numbers of patients randomized and analyzed are necessary for interpreting the trial's findings [4].

Published reports of trials, and research, in general, must be an unbiased and accurate description of their conduct and findings [5]. Although the Consolidated Standards of Reporting Trials (CONSORT) statement has helped improve transparency in reporting methods and findings from RCTs [1],[3],[6], it has not been possible to identify the degree to which published reports accurately reflect the trial conduct. One of the areas that has not been well examined is how accurately trial publications describe the numbers of participants randomized and how they were analyzed. Existing research on consistency of criteria for including participants in the analysis and transparency in reporting focuses on two types of analysis, ITT and modified ITT (MITT) [7]–[13]; this research compares either the types of analysis across publications or what was planned in the trial protocol with what was reported in the publication [8]. We are unaware of research comparing what actually happened in the trial (i.e., patients included in internal study analyses) with what was reported in the trial publication.

Our objective was to compare, using a sample of industry-sponsored trials, the transparency and accuracy of reporting the numbers of participants, the description of types of analyses, and the definitions of each type of analysis (i.e., criteria for including participants in the analysis).

Methods

Data Sources

To achieve our study objective, we compared the description of the types of analysis and their definitions in the internal company protocol and research report (i.e., what was planned and what was done) to the corresponding publication (i.e., what was reported). The internal company documents used in our study were made available as part of litigation against Pfizer, Inc., (the defendants), by Kaiser Foundation Health Plan, Inc., (the plaintiffs), in Boston, Massachusetts [14]. We conducted our study as a scholarly pursuit independent of the litigation.

All trials included in our study were sponsored by Pfizer and Warner-Lambert's subsidiary Parke-Davis (Pfizer and Parke-Davis) and related to four off-label uses of gabapentin (i.e., indications not approved by the US Food and Drug Administration [FDA]) [15],[16]: migraine prophylaxis, and treatment of bipolar disorders, neuropathic pain, and nociceptive pain. We examined internal company documents and all trial publications that were made available to one of the authors (KD) for preparation of a report for the plaintiffs' lawyers as part of the litigation against Pfizer [15]. One additional trial publication was identified through an ad hoc bibliographic search [17]. Publications for trials included in our sample were dated between 1998 and 2008 in most cases [17]–[26], except for one trial publication in 1987 [27]. When there was more than one publication emanating from a trial, we specified one main publication for each trial on the basis of the following order of priority: full-length journal article; letter to the editor with study results; non-systematic review with pooled analysis; conference abstract.

For each trial, we examined internal company protocols, including amendments, and statistical analysis plans (SAPs), research reports, and the main publications. We used the label described in the internal company document to identify whether it was a protocol, SAP, or a research report. Internal company documents labeled “research report” were produced by or for Pfizer and Parke-Davis and included a detailed description of the methods, results of all statistical analyses in the trial, and a discussion. Research reports typically also included study protocols, protocol amendments, and SAPs in the appendices. We considered internal company documents labeled “Protocol” as the study protocol and those labeled “Inferential analysis plan,” “Statistical analysis plan,” or “Report and analysis plan” as the SAP.

We identified one main publication for 13 of 21 trials for the four indications [17]–[29]. We included in our analysis 11 published RCTs. We excluded one published trial because we did not have access to any internal company documents [28] and another published trial because it was not randomized and is thus, not applicable to comparisons we made for the types of analysis [29]. In addition, we excluded eight trials that were never published.

Data Extraction and Comparisons

From the research report and the publication, we extracted data on participant flow (i.e., number of participants assessed and excluded for eligibility at screening, number randomized to treatment groups, number that received and did not receive the assigned treatment, number lost to follow-up, discontinuing intervention, and number included in the efficacy [per the publication-specified primary outcome] and safety analyses).

From the protocol, research report, and publication, we extracted data on the types of analysis described for the efficacy and safety outcomes, and the definition for each type of analysis (i.e., criteria for including participants in the analysis).

We made the following comparisons: (1) numbers of participants randomized and analyzed for efficacy (publication-specified primary outcome): research report versus publication (ten trials where both documents were available). (2) types of analyses for efficacy and safety, and their definitions: protocol versus research report (eight trials where both documents were available); protocol versus publication (nine trials where both documents were available); research report versus publication (ten trials where both documents were available).

One author (SSV) extracted data into an MS Access database and a second author (TL) verified all of the extracted data, by comparing it with the original documents. There was 97.5% agreement between the two authors (SV and TL) on data extracted from protocols and SAPs and 73% agreement on data extracted from research reports and publications. A third author (KD) independently verified, through reference to the original documents, all discrepancies between the first two abstractors and a 10% random sample of items (sample taken using a random number table) agreed upon by the first two abstractors (complete agreement for the 10% sample). We resolved disagreements among authors through discussion and consensus.

Results

Reporting on Numbers of Participants Randomized and Analyzed: Comparing Research Reports Versus Publications (Ten Trials Where Both Documents Were Available)

Number randomized

There was disagreement between the research report and the main publication on the number of participants randomized for three of ten trials (see Table 1). For one of the three trials, the research report described the completed study findings while the main publication was a conference abstract describing preliminary results [27]. We were unable to identify a reason for the disagreement in the remaining two trials [24],[26].

Number analyzed

For six of ten trials, we were unable to compare the research report with the main publication for the number of participants analyzed for efficacy, because the research report either did not describe the publication-specified primary outcome or did not describe the publication-specified type of analysis (see Table 1 footnotes). There was agreement for four of ten trials.

Because of substantial variations in terminology within documents related to the same trial as well as across trials, we decided not to make comparisons between the research report and the main publication on numbers related to participant flow. We made this decision after two authors (SV and TL) independently attempted to reconcile variations in terminology followed by a consensus discussion among all authors.

Two examples of variation in terminology within documents from the same trial follow. For study 945-220, the research report listed 24 patients in the gabapentin group and nine in the placebo group under the heading “Premature Discontinuation”; and the main publication specified that 24 patients in the gabapentin group and nine in the placebo group were “Withdrawn” and provided “Reasons for Discontinuation.” [23]. For study 945-276, the research report specified that 21 patients in the gabapentin group and ten in the placebo group “dropped out” but did not provide a definition for “drop-outs”; the main publication described that 21 patients in the gabapentin group and nine patients in the placebo group “Discontinued Intervention.” [20]

Variation in terminology may affect study findings, for example in study 945-220, MITT was described in the publication as follows: “This population included any patient who was randomized, took at least one dose of study medication during SP2 [Stabilization Period 2], maintained a stable dose of 2,400 mg/day during SP2, had baseline migraine headache data and at least 1 day of migraine headache evaluations during SP2” [23].

If “Premature discontinuation” had meant that participants discontinued the assigned treatment but outcomes were still assessed during “stabilization period 2” of follow-up then they would be eligible for inclusion in the MITT analysis described in the publication.

In contrast, if “Premature discontinuation” had meant that the participants were no longer followed up for assessing outcomes then they would not be eligible for inclusion in the MITT analysis (in all cases where the “premature discontinuation” resulted in no outcomes being assessed during stabilization period 2).

Thus, the two interpretations would mean that different subsets of participants are included in the analysis, even while using the same definition for the type of analysis, thereby leading to different findings.

Reporting on the Types of Analysis for Efficacy and Safety: Comparing Protocols Versus Research Reports (Eight Trials Where Both Documents Were Available)

We observed extensive variability and inconsistency (i.e., disagreements) among documents (i.e., protocol, research report, SAPs, and publication) within the same trial and across trials for the types of analyses that were specified as well as their definitions. An ITT analysis was specified in the protocol for five of eight trials and in the research report for seven of eight trials. Both documents described a definition for ITT in three of eight trials and there was a disagreement in the definition for two of three trials (see Table 2). An example of a disagreement in the definition is shown below.

Study 945-224

The protocol defined an ITT analysis as follows: “…this includes all patients randomized to treatment who received at least 1 dose of study medication.”

The methods section in the research report for the same trial defined ITT as follows: “The ITT population was defined as all patients randomized who received at least 1 dose of study medication in the double-blind phase.”

The results section in the research report defined ITT as follows: “This population comprised all patients who received at least 1 dose of study medication and who had an observation for the primary efficacy parameter at baseline.”

We also found disagreements in the definition for safety analysis between the protocol and research report for the same eight trials. The definition for a safety analysis was described in both documents in the case of five of eight trials; there was disagreement on the definition for two of five of these trials (see Table 2 and Text S1).

Reporting on the Types of Analysis for Efficacy and Safety: Comparing Protocols Versus Publications (Nine Trials Where Both Documents Were Available)

Seven different types of analyses for efficacy were described in protocols (including amendments and SAPs) and publications across the included trials (see Table 3). The definitions for the types of analysis were not always consistent across trials or between the protocol and publication for the same trial.

Six different definitions for ITT were described in the protocols, SAPs, and publications we examined (see tables 4 and 1 in Text S1). In neither type of document did the definition for ITT ever match its widely accepted description (i.e., all randomized patients in the groups they were assigned) [1]–[3]. For three of nine trials, a definition was available in both the protocol and publication; there was disagreement for two of the three trials. An example of a disagreement in the definition is shown below.

Study 945-209

The protocol defined ITT as follows: “A secondary population will be the Intent-to-Treat population which is defined as all patients randomized to treatment and who have at least 1 postrandomization visit.”

The publication defined ITT as follows: “The efficacy analyses were carried out on the intent-to-treat (ITT) population that included all randomized patients who received at least one dose of study medication.” [24]

There were disagreements between the protocol and publication for other types of analysis for efficacy, including “modified intent to treat (MITT),” “per protocol,” “evaluable,” “evaluable for efficacy” and “Population to be analysed” (see Table 5).

Criteria for including participants in analyses of safety outcomes were variably described in the protocols and publications across trials. Five different definitions for the safety analysis were used in protocols, SAPs, and publications across all included trials (see Tables 2 and 6). For two of nine trials, the definition for safety analysis was described in both documents and there was disagreement on the definition in one of the two trials.

Reporting on the Types of Analysis for Efficacy and Safety: Comparing Research Reports Versus Publications (Ten Trials Where Both Documents Were Available)

Agreement between the research report and publication on description of types of analysis for efficacy varied across the trials included for this comparison. For example, while a definition for ITT was available in both the research report and publication in four trials there was disagreement for two of the four trials. There was also frequent disagreement between the research report and publication for other types of analyses for efficacy (data shown in Text S1).

The research reports described criteria for including participants in analyses of safety more often than publications, and in greater detail. For four of the ten trials, a definition for safety analysis was described in both documents; there was disagreement for one of the four trials.

Nature of Disagreements across the Trial Documents

The most commonly observed form of disagreement on types of analyses across the documents was omission from the publication of one or more types of analysis for efficacy specified in the trial protocol (five of nine trials with both documents) or research report (seven of ten trials with both documents).

Discussion

In our sample of industry-supported trials in off-label uses of gabapentin, we observed discrepancies between what was reported in trial publications and what was described in internal company research reports. In this regard, we found that the trial publication was not a transparent, or accurate (presuming that the research report truly describes the facts), record for the numbers of participants randomized and analyzed for efficacy. In three of ten trials in our sample, the number of participants randomized in the trial, as specified in the “main publication” [24],[26],[27], was not the same as that described in the research report. The “main publication” was a full-length journal article for two of the three trials with a disagreement in the number of participants randomized in the trial [24],[26], and a conference abstract describing preliminary results for the third trial [27]. In one case, the description in the publication did not include data from 40% of participants actually randomized in the trial (as described in the research report; see Table 1) [26]. There was such wide variation in describing the participant flow in the included trials, even among documents for the same trial, that we were unable to summarize what we found.

In addition, we observed extensive variability, both among documents for the same trial and across trials, in the types of analyses specified for efficacy and safety as well as in the criteria for including participants in each type of analysis; this is consistent with Chan's findings comparing protocols and publications [8]. Our study extends comparisons of the protocol and publication to comparisons of analyses presented in internal company research reports and analyses presented publicly.

We are concerned that, even for commonly used types of analysis such as ITT, a number of different definitions were used across trials included in our sample. Trial findings may be sensitive to the alternative definitions used for the type of analysis, i.e., analyses using different definitions of the type of analysis can include different subsets of participants, thereby leading to different findings.

Because internal company documents are not generally available, our study provides a unique snapshot of how well a publication may reflect what is known by the company. Since doctors rely on publications for information about a drug's effectiveness for off-label indications, our study raises particularly important questions that may be applicable more broadly to other drugs like gabapentin, which has been prescribed so frequently for off-label indications [30].

Because we included a small number of trials for off-label indications sponsored by a single company using documents released as part of litigation, our findings may not apply to other industry-sponsored trials, to trials for other off-label indications, or to trials sponsored by other entities. We did not undertake a systematic literature search to identify any additional publications that may have described results from the included trials. In nearly every case, since the documents associated with each trial were obtained as part of the legal discovery process, we believe that we have a complete list of publications for each trial.

We did not have access to the protocol for two trials and the research report for one trial, and were unable to compare them with the corresponding trial publication. For one trial (study 879–201), we used a conference abstract describing preliminary results [27] as the “main publication” for our comparisons with the protocol and research report. Because the only information the public has about the findings from study 879-201 is from the conference abstract [27], we believe that our observations using it are relevant in the context of our study. We are not able to determine the cause for missing data and inadequate reporting of data for any of the trial documents we examined. For example, editing to meet space constraints may have been responsible for omission of information from published reports.

In addition, we did not assess whether disagreements in descriptions of types of analyses resulted in different participants being analyzed for efficacy and safety. We also did not examine the impact of the observed disagreements on the effectiveness of gabapentin for the indications specified in our study, as meaningful assessment would have required a systematic review of each trial topic. We encourage this important area of research, however.

The discrepancies in description of types of analyses we observed in our study are not unique to trials sponsored by for-profit entities such as pharmaceutical companies. Using trial protocols obtained from institutional review boards and corresponding publications, previous research has shown that disagreements such as those we described in our study can be observed in trials with funding from both for-profit and not-for-profit entities [8].

Our findings support recommendations that simply stating in the publication that an analysis used ITT is not adequate [2],[3]. Among the trials that reported a planned ITT analysis (about half of the trials in our analysis) in the protocol, the definition of ITT (i.e., which participants were included in the ITT analysis) varied. In no case did the documents describe an ITT analysis according to the widely accepted definition, i.e., all randomized participants in the groups to which they were assigned. Because ITT is recommended for the primary analysis of randomized trials [1]–[3], modifying its definition, whether across studies or within a study, defeats the purpose of a standard.

Even if the definition of an ITT is “all randomized participants analyzed as part of the group to which they were assigned,” results from a single trial can still vary depending on the method used to impute or account for missing data (e.g., patients who missed a visit). Indeed, a recent study showed that even among reports of methodological studies, the definition of ITT varied with regard to how missing outcome data are handled in the analyses [31]. A proposed revision to the CONSORT statement reportedly will include a requirement for describing any imputation of missing data used to conduct an ITT analysis [32].

On the basis of our findings, we suggest that the following items be considered for inclusion as recommended standards for reporting trials in subsequent revisions to the CONSORT statement: (1) explicitly state all protocol-specified type(s) of analysis, or the analysis populations, for analyses of primary, secondary, and safety outcomes; (2) clearly state the criteria used to define each type of analysis, if more than one is specified; (3) explicitly specify the numbers of participants who do not meet each of the stated criteria, and are therefore excluded, for each type of analysis; (4) specify terminology for consistent use across trials, for example, for types of analysis, participants who do not receive treatment as assigned (“treatment dropouts”), and patients not included in the analysis (“analysis dropouts”) [4]; (5) recommend that the individual patient data used for all analyses described in the trial report be made accessible to the biomedical community along with the published article.

In addition to being relevant to revisions of the CONSORT statement, the recommendations listed above are also relevant to the Standard Protocol Items for Randomized Trials (SPIRIT) Initiative, which includes items on methods to handle protocol deviations and missing data but does not explicitly require a detailed definition for the type of analysis for efficacy and safety [33].

Although revisions to reporting standards can contribute to improving transparency of reporting, they cannot assure accuracy of reporting as noted above. A variety of initiatives could improve the situation, although there is no way to completely eliminate deliberate fraud. Transparency, through public availability of trial protocols, research reports, and results, is a possible solution [34]. But existing initiatives to this end do not go far enough to ensure transparency.

Our work emphasizes that research reports, which are similar to “clinical study reports” described in another context [35], provide a more comprehensive and presumably, a more accurate narrative of the conduct of a trial and its findings than a journal publication. For example, the research reports for trials in our sample more often provided details about the definitions for types of analyses, while the corresponding trial publications omitted such details. The research reports are typically considered confidential documents and are rarely available to the public or the wider biomedical community. Our findings lend support to the principle that such “research reports” or “clinical study reports” must be readily accessible to the biomedical community and the public [35] so that the veracity of what is reported in the publications can be assessed.

Current US legal requirements for reporting study findings are inadequate both in scope and detail. FDA approval is not required for off-label use of drugs. Although information on trials in off-label uses of drugs may be submitted to the FDA, the agency's reviews of the trial data are made publicly available only if the application to market the drug is approved [36]–[38]. Consequently, for trials of off-label uses of drugs that are not submitted to the FDA, or that are submitted to the FDA as part of a withdrawn or unsuccessful application to market the drug, the only publicly available source of information about effectiveness accessible to most decision-makers is the published literature.

The FDA Amendments Act of 2007 requires that summary results for trials of off-label uses of approved drugs, even some investigational new drug-exempt trials, must be posted on ClinicalTrials.gov, for all trials not completed before 2007 [39],[40]. Having results of trials in off-label uses available publicly allows comparison of the sponsor's trial data with information in publications [41]–[43]. It is not clear to us, however, that having summary data available through ClinicalTrials.gov is sufficient to allow confirmation of the trial findings by other investigators. Certainly, the final dataset along with the full clinical study report from industry-sponsored trials for approved drugs should be made available by the FDA to the public for this purpose [44]. The European Medicines Agency (EMA), in contrast, makes publicly accessible its summary (i.e., European Public Assessment Report) and certain documents submitted to the agency as part of applications requesting authorization to market drugs [35],[45]–[47]. Other investigators have successfully obtained documents from the EMA, including clinical study reports, to verify claims in publications about a drug's effectiveness [35].

Our findings highlight the need for standardizing the definitions for various types of analyses that may be conducted to assess intervention effects in clinical trials, delineating the circumstances under which different types of analyses are meaningful, and educating those who are involved in conducting and reporting trials such that the standards are consistently adopted. We believe that our findings lend support to policy considerations such as extending mandatory registration to include all clinical trials [48],[49], making full trial protocols and trial data publicly available through trial registration or other means, and ensuring that regulations pertaining to compulsory reporting of results apply both to trials conducted for regulatory authority-approval and to trials in off-label indications of interventions. Health interventions are administered to members of the public on the basis of trial findings. It is time for the balance of power in access to information from clinical trials to be shifted from those sponsoring the trials to the public at large.

Supporting Information

Attachment 1