Development of the Survey Instrument and Study Oversight

Table 1. Table 1. Characteristics Used to Differentiate Trials of Hypothetical New Drugs, According to Methodologic Rigor.

We developed hypothetical scenarios in which three new drugs were being evaluated for the treatment of disorders commonly encountered in primary care: hyperlipidemia, diabetes, and angina. “Lampytinib” would be used for dyslipidemia in patients who had unacceptable side effects from statins, “bondaglutaraz” would be used for diabetes and low levels of high-density lipoprotein cholesterol in patients who were taking metformin and a sulfonylurea and who were unwilling or unable to add insulin, and “provasinab” would be used for angina in patients with untreatable multivessel coronary disease who were taking maximal doses of beta-blockers. Since internists report that they frequently read only the abstracts when reviewing the medical literature,23 we created abstracts describing trials of these drugs in which we varied the drug being tested, the trial's methodologic rigor, and the funding source. For each drug, one trial had a high level of rigor, one had a medium level of rigor, and one had a low level of rigor. The features defining these levels were based on guidelines24,25 and on our experience in conducting randomized trials26,27 and in studying evidence-based drug-evaluation and prescribing practices28,29 (Table 1). Differences in methodologic rigor among the trials were consistent across drugs. All the trials had similar effect sizes, and statistically significant results.

We then added a variable describing the trial's funding status. Each abstract included one of three variations: no funding source mentioned, funding by the National Institutes of Health (NIH), or funding by a pharmaceutical company, with financial involvement in that company on the part of the lead author. (Companies named in the disclosure statements were randomly selected from the top 12 global pharmaceutical enterprises.30)

Assigning one of the three conditions to each of the three variables (drug, study design, and funding source), we created 27 different abstracts describing a hypothetical new-drug trial. The abstracts, along with the study protocol, are provided in the Supplementary Appendix, available with the full text of this article at NEJM.org. The survey was pretested among 12 physicians certified in internal medicine.

The study was approved by the institutional review board at Brigham and Women's Hospital, with written informed consent implied by the participant's completion of the survey. There were no agreements concerning confidentiality of the data between the authors and the institutions providing financial support for the study.

Survey Sample

The American Board of Internal Medicine (ABIM) maintains a list of diplomates with active certification and maintenance of certification. The data set included demographic characteristics and information about medical training and responses to the ABIM Practice Characteristics Survey.31,32 From a potential sample of 45,398 physicians, we randomly identified 514 who reported spending 40% or more of their time and 21 hours or more per month in activities related to patient care and spending 50% or less of their time in the intensive care unit, the emergency department, or the cardiac catheterization laboratory. Of these possible participants, 11 had noncurrent contact information, resulting in a potential sample of 503 physicians.

Survey Administration

Between July 2011 and October 2011, physicians in the sample received two postcards and three e-mail messages from ABIM indicating that they had been randomly selected to participate in a study investigating how physicians make prescribing decisions. These communications included the names of the sponsoring institutions and the lead investigators, a link to the online survey, an opportunity to opt out, and an offer of a $50 honorarium on completion of the survey (see the Supplementary Appendix). Physicians who did not respond were mailed a printed version of the invitation along with a $5 bill and an offer of $45 on completion of the survey. Those who still did not respond received a telephone reminder.

Participants were presented with three abstracts, each of which described a trial of a different hypothetical new drug. Participants were told to assume that the hypothetical drug had recently been approved by the Food and Drug Administration and was covered by insurance and that the abstract was from a “high-impact” biomedical journal and written by academic physicians at established U.S. universities. We randomly varied the level of methodologic rigor and the disclosure so that the three abstracts that the physicians received described trials at each level of methodologic rigor and with each disclosure variation.

Dependent Measures and Variables

Each abstract was followed by the same set of questions, with the choice of answers based on a 7-point Likert scale. Physicians were asked about their likelihood of prescribing the new drug (ranging from a score of 1 for “very unlikely to prescribe” to a score of 7 for “very likely to prescribe”), the level of methodologic rigor of the trial (ranging from a score of 1 for “not at all rigorous” [low rigor] to a score of 7 for “very rigorous” [high rigor]), and their confidence in the study investigators' conclusions (ranging from a score of 1 for “not confident at all” to a score of 7 for “very confident”). Secondary outcomes were participants' rating of the importance of the trial20 and their level of interest in reading the full article. (These questionnaires are provided in the Supplementary Appendix.)

After reading the third abstract, participants were asked to estimate how many abstracts describing trials of medications they had read in biomedical journals in the past month and to respond to the following question: “Do you think that pharmaceutical company funding is likely to influence the outcome of scientific studies about the efficacy and safety of pharmaceuticals in favor of the drug in question?” Finally, participants were asked about any financial support they had received from drug, device, or other medically related companies in the previous year.33

Statistical Analysis

For each question, we estimated a hierarchical proportional-odds regression model, using the appropriate Likert-scale response as the outcome. This model included a random intercept for each physician to account for within-physician correlation of responses across abstracts, as well as fixed effects for methodologic rigor (low, medium, or high), disclosure statement (industry funding, NIH funding, or no statement), and drug. A second model included interactions between the indicators for funding source and methodologic rigor.34 Because we developed the three hypothetical drugs solely to obtain data about the effects of the level of methodologic rigor and variation of disclosure, we did not separately analyze differences among the drugs.

To investigate the association between characteristics of the physicians and survey responses, we created proportional-odds models that included random intercepts for physicians; indicators for the variables describing methodologic rigor, type of disclosure statement, and drug; and terms for physician characteristics (i.e., age, sex, location of medical school [U.S. vs. non-U.S.], type of practice [general internal medicine vs. subspecialty], proportion of time spent in clinical care [≥80% vs. <80%], hours per month spent in clinical care [≥80 vs. <80], acceptance of gifts from industry [any vs. none], and opinion about whether industry funding influences the outcome of trials in favor of the drug being tested [a score of 6 or higher vs. a score of 5 or lower on the 7-point scale, with higher numbers indicating a perception of a stronger influence]).