Publication bias in clinical trials

Last week the New England Journal of Medicine (NEJM) published a paper on selective outcome reporting in clinical trials (Vedula et al. 2009). The primary and secondary outcome(s) of a clinical trial could for example be survival in cancer patients or rate of heart attacks and other cardiovascular events in patients taking cholesterol-lowering drugs. These outcomes are defined in the study protocol before the first patient is treated, and whether or not the primary outcome is reached (using statistical testing) defines whether a trial was positive or negative. The study protocol is approved by an institutional review board (IRB) and can only be changed later (and that includes changes in the protocol-defined outcomes) if again approved by the IRB.

The study in the NEJM examined the outcomes of 20 clinical trials for the drug gabapentin using internal documents of the drug companies Pfizer and Parke-Davis, and compared them to the outcomes reported for those trials that were published as peer-reviewed papers. The internal company documents were obtained as the result of litigation in which the company admitted guilt for off-label marketing (i.e. marketing for uses that were not approved by the Food and Drug Administration) of gabapentin.

12 of the 20 trials were published as peer-reviewed papers. In 8 of these 12 papers the primary outcome differed from the primary outcome defined in the protocol. New primary outcomes were introduced in 6 papers, and protocol-defined primary outcomes were not reported in 5 papers. Trials with primary outcomes that were not significant were either not reported as full paper or were reported with a changed primary outcome.

Selective outcome reporting has also been reported by other authors and is not limited to trials funded by drug companies (Chan 2004). Selective outcome reporting is a major problem because it a) distorts our scientific knowledge and b) is unethical as it involves research on human subjects. This could for example lead to repeated studies of a clearly ineffective or harmful drug or intervention. The distortion of our scientific knowledge by selective outcome reporting in scientific journals is also a concern for research in other areas. Confirmation of important research findings by independent groups is important, but a lot of research is probably repeated simply because negative results were not published. Some of these reporting biases could be avoided by making more unpublished research data available, either by Open Notebook Science or by publishing “unexciting negative” findings in peer-review journals or preprint archives such as Nature Precedings.

Clinical medicine tries to solve the problem of publication bias by making public registration of clinical trials mandatory before patient enrollment. This registration is required by most major medical journals since 2004 (De Angelis 2004). U.S. legislation also requires clinical trial registration and since 2008 (FDAAA: Push to open data in clinical medicine) this includes outcome reporting within 12 months after data for the last subject were received in the publicly available Clinicaltrials.gov database. These efforts should ensure ethical standards of clinical research, and help to avoid the kinds of biases reported in the paper by Vedula et al.

Clinical trial registries obviously also serve other purposes, as they help interested patients and their relatives and treating physicians to find clinical trials they want to participate in, and they help researchers and clinicians to do a systematic overview of the ongoing research in a particular field. Just as there are centralized databases (e.g. PubMed Central and the Current Controlled Trials metaRegister of controlled trials) and institutional repositories for full-text papers, there are also both central (e.g. Clinicaltrials.gov – Both PubMed Central and Clinicaltrials.gov are efforts by the U.S. National Institutes of Health) and institutional clinical trial registries. But in contrast to institutional repositories, there are no standard software tools available that an institution can use. I am involved in building a clinical trial registry for our institution, and the effort is both technically demanding, and socially challenging. The information in a clinical trial registry is constantly changing and we need to keep the efforts required by the individual researchers at a mininum. We also have to walk a fine line of what information can be made publicly available, and here we follow the standards provided by both Clinicaltrials.gov and the WHO.

EudraCT, the mandatory European clinical trials registry is regretably not available to the public.

References

Vedula, S., Bero, L., Scherer, R., & Dickersin, K. (2009). Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use New England Journal of Medicine, 361 (20), 1963-1971 https://doi.org/10.1056/NEJMsa0906126

Chan, A.-W. (2004). Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Canadian Medical Association Journal, 171(7), 735–740. https://doi.org/10.1503/cmaj.1041086

De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., … Weyden, M. B. V. D. (2004). Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors. New England Journal of Medicine, 351(12), 1250–1251. https://doi.org/10.1056/nejme048225