Yusuke Tsugawa , research associate 1 2 3, Anupam B Jena , Ruth L Newhouse associate professor, and faculty research fellow 4 5 6, E John Orav , associate professor 2 7, Ashish K Jha , K T Ki professor of international health, and director 12 3 8 1Department of Health Policy and Management, Harvard T H Chan School of Public Health, Boston, MA, USA 2Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA, USA 3Harvard Global Health Institute, Cambridge, MA, USA 4Department of Health Care Policy, Harvard Medical School, Boston, MA, USA 5Department of Medicine, Massachusetts General Hospital, Boston, MA, USA 6National Bureau of Economic Research, Cambridge, MA, USA 7Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA 8VA Boston Healthcare System, Boston, MA, USA Correspondence to: Y Tsugawa ytsugawa{at}hsph.harvard.edu Accepted 3 January 2017

Abstract Objective To determine whether patient outcomes differ between general internists who graduated from a medical school outside the United States and those who graduated from a US medical school. Design Observational study. Setting Medicare, USA. Participants 20% national sample of data for Medicare fee-for-service beneficiaries aged 65 years or older admitted to hospital with a medical condition in 2011-14 and treated by international or US medical graduates who were general internists. The study sample for mortality analysis included 1 215 490 admissions to the hospital treated by 44 227 general internists. Main outcome measures Patients’ 30 day mortality and readmission rates, and costs of care per hospital admission, with adjustment for patient and physician characteristics and hospital fixed effects (effectively comparing physicians within the same hospital). As a sensitivity analysis, we focused on physicians who specialize in the care of patients admitted to hospital (“hospitalists”), who typically work in shifts and whose patients are plausibly quasi-randomized based on the physicians’ work schedules. Results Compared with patients treated by US graduates, patients treated by international graduates had slightly more chronic conditions. After adjustment for patient and physician characteristics and hospital fixed effects, patients treated by international graduates had lower mortality (adjusted mortality 11.2% v 11.6%; adjusted odds ratio 0.95, 95% confidence interval 0.93 to 0.96; P<0.001) and slightly higher costs of care per admission (adjusted costs $1145 (£950; €1080) v $1098; adjusted difference $47, 95% confidence interval $39 to $55, P<0.001). Readmission rates did not differ between the two types of graduates. Similar differences in patient outcomes were observed among hospitalists. Differences in patient mortality were not explained by differences in length of stay, spending level, or discharge location. Conclusions Data on older Medicare patients admitted to hospital in the US showed that patients treated by international graduates had lower mortality than patients cared for by US graduates.

Introduction International medical graduates make up a quarter of the physician workforce in the United States, United Kingdom, Canada, and Australia.12345 In the UK, these graduates are largely from India, Republic of Ireland, and Pakistan, and in the US, largely from India, the Philippines, and Pakistan (not including US citizens who have gone abroad for medical education and returned to the US to practice).3 The licensure process for international medical graduates varies between these countries: in the UK, students have to pass an English language and Professional and Linguistic Assessments Board tests so they can register with the UK General Medical Council to practice.6 In the US, students must pass two examinations that test medical knowledge and one examination that assesses clinical skills, and they must complete accredited residency training in the US (see supplementary appendix 1 for details).7 Although international graduates are required to pass these examinations to practice medicine in the UK and US, there have been no UK based or US based standardized accreditations of foreign medical schools, and consequently both policymakers and the public have expressed concerns about the quality of care provided by these graduates.891011 In addition, studies have suggested that there may be implicit biases that the quality of care international graduates deliver may be inferior to that of doctors trained domestically, potentially leading to discriminations against physicians who have trained abroad.121314 In response, in the US, the Educational Commission for Foreign Medical Graduates recently announced the requirement for all foreign medical schools to be accredited through a formal process by 2023.15 However, it is unclear whether current licensure processes for international medical graduates provide adequate safeguards for ensuring high quality care. Though concerns about the quality of care delivered by international medical graduates are often raised, available data comparing patterns of care and outcomes between international medical graduates and doctors trained domestically are limited. The evidence that does exist is inconsistent and has focused on measures such as test scores161718 and process measures of quality.19 The studies that have examined patient outcomes have been small and from a few US states or a single Canadian province, making it difficult to draw conclusions about the quality of care provided by international graduates more broadly.20212223 Given the substantial public interest in and ongoing concerns about the quality of care such graduates provide in the US, UK, and other high income countries891011 and policymakers’ efforts to assure more consistency in foreign medical education, it would be helpful to have empirical data on how international medical graduates perform. Using a national sample of elderly patients admitted to hospital in the US, we sought to answer three key questions. First, do patients treated by internists who graduated outside the US have different mortality compared with patients treated by internists who graduated from a US medical school? Second, given substantial efforts and interest in reducing rates of readmissions and costs of care, do international graduates have higher readmission rates or provide more costly care than US graduates? Finally, do differences in patient outcomes and costs of care between the two types of graduates, if any, vary by clinical condition?

Methods We linked four data sources: the 100% Medicare Inpatient Files (2011-14), the 20% Medicare Carrier Files (2011-14), the American Hospital Association annual survey on hospital characteristics, and a comprehensive physician database collected by Doximity (see supplementary appendix 1 for details of this database, including its validation242526). We identified Medicare fee-for-service beneficiaries age 65 years or older who were admitted to the hospital with a medical condition during 1 January 2011 to 31 December 2014, as defined by the presence of a Medicare severity-diagnosis related group on hospital admission. The unit of analysis was hospital admissions. Our sample was restricted to patients admitted to the hospital who were treated in acute care hospitals and excluded elective admissions and those in which a patient left against medical advice. To allow sufficient follow-up periods, we excluded patients admitted in December 2014 from analyses of 30 day mortality and patients discharged in December 2014 from 30 day readmission analyses. For the cost analyses, we restricted our sample to patients with admission and discharge dates in 2011-2014 to ensure that we observed all costs during hospital stay. We assigned each hospital admission to a physician based on the national provider identifier in the carrier file that accounted for the most Medicare part B spending during that hospital stay.262728 In the US, costs for care of patients admitted to hospital consist of part A spending (payments to hospitals) and part B spending (professional and other fees determined by the physician). We focused on part B spending because it encompasses professional and other services at the discretion of physicians; and for patients admitted to hospital, part A spending is largely invariant to physician decisions because of the fixed payment by Medicare severity-diagnosis related group. On average, 51.1%, 22.0%, and 11.1% of total part B spending was accounted for by the first, second, and third highest spending physicians, respectively. We focused on general internists to avoid comparing physicians across different specialties. Physicians’ specialties were identified using multiple sources, including the data collected by the American Board of Medical Specialties and other specialty societies. We also restricted our analysis to graduates of allopathic medical schools. Identification of international medical graduates The Doximity database includes information on the medical school from which a physician graduated. We identified countries where each medical school was located using multiple sources, including the International Medical Education Directory organized by the Foundation for Advancement of International Medical Education and Research,29 the Association of American Medical Colleges database,30 and web pages of individual medical schools. In our study sample, data on graduate medical schools were available for approximately 85% of general internists. Patient outcomes The primary outcome was 30 day mortality of patients. Secondary outcomes were 30 day readmission rates and costs of care. We defined the costs of care as total part B spending for each hospital admission. Adjustment variables We adjusted for patient characteristics, physician characteristics, and hospital fixed effects (ie, hospital specific indicator variables included as covariates). Patient characteristics included age in five year increments (65-69 years through 90-94 years, and ≥95 years), sex, race or ethnic group (non-Hispanic white, non-Hispanic black, Hispanic, other), primary diagnosis (defined by Medicare severity-diagnosis related group), Elixhauser comorbidity index31 (27 coexisting conditions), median household income estimated from residential zip codes (in deciles), an indicator for dual Medicare-Medicaid coverage, year indicators, and day of the week. Physician characteristics consisted of age in five year increments (<35 years, 35-39 years, and so on through 65-69 years, and ≥70 years), sex, and patient volume (number of Medicare beneficiaries treated as a continuous variable with quadratic and cubic terms to allow for a non-linear relationship). Hospital fixed effects account for both measured and unmeasured characteristics of hospitals, allowing us to effectively compare patient outcomes between international and US medical graduates within the same hospital.323334 Statistical analysis We first examined whether patient mortality differed between international and US medical graduates using three models. Model 1 compared patient outcomes between the two graduate groups, adjusting for patient characteristics using a multivariable logistic regression model. Model 2 adjusted for all variables in model 1 plus physician characteristics, thereby comparing adjusted patient outcomes of the international graduates with US graduates across hospitals. Model 3 adjusted for all variables in model 2 plus hospital fixed effects, effectively comparing international graduates with US graduates within the same hospital.323334 We clustered standard errors at the physician level.35 We then evaluated whether readmission rates and costs of care differed between the international and US graduates. For the readmission analyses, we used the same set of multivariable logistic regression models as we did for the mortality analyses. For our analyses on costs, we used multivariable ordinary least squares regression models, with the same set of adjustment variables as those used for the analyses of mortality and readmissions. Because we evaluated three outcomes, we considered a P value of less than 0.016 to be statistically significant. Finally, we assessed whether differences in patient outcomes varied according to the primary condition for which a patient was admitted. We evaluated six major conditions treated by general internists, selected based on frequency: sepsis, pneumonia, congestive heart failure, chronic obstructive pulmonary disease, urinary tract infection, and arrhythmia (see supplementary table A for a list of ICD-9 (international classification of diagnosis, ninth revision) codes for these conditions). Sensitivity analyses We conducted sensitivity analyses. First, to address the possibility that the international graduates might treat patients with lesser or greater unmeasured severity of illness, we restricted the study population to patients treated by physicians who specialize in the care of patients admitted to hospital (“hospitalists”) (we focused on general internists (including both hospitalists and non-hospitalist general internists) for main analyses, and hospitalists for the sensitivity analysis). These physicians typically work in shifts, and therefore, within the same hospital, patients treated by them are plausibly quasi-randomized to a given physician based on that hospitalist’s work schedule.2636 Second, since there are multiple ways to assign physicians to patients, we tested two alternative methods to assign physicians to patients: assigning physicians who had the largest number of evaluation and management claims, and assigning physicians who billed the first evaluation and management claim for a given hospital admission.37 Third, to account for the influence of international medical graduates who were US citizens, we excluded international graduates who graduated from medical schools in Central America and the Caribbean, because three quarters of US citizen international medical students graduate from medical schools in these countries.113839 Fourth, since differences in length of stay, utilization of care (total part B spending per hospital admission), or discharge location might explain differences in patient outcomes between the graduates, we further adjusted our regression models for these variables. Fifth, to deal with the impact of unobserved care preferences of patients, we excluded patients with cancer and patients who were discharged to hospice care.40 Sixth, as international graduates might be more or less likely to work as intensivists in intensive care units, we excluded hospitals with a medical intensive care unit. Seventh, it is possible that residents bill Medicare claims on behalf of their attending physicians at teaching hospitals, and differences in patient outcomes might be related to the quality of care delivered by residents. To address this, we stratified our sample by teaching status of hospitals (major teaching, minor teaching, and non-teaching hospitals), and within each group we compared international medical graduates with US medical graduates (adjusted for patient and physician characteristics and hospital fixed effects). Finally, we examined whether patient outcomes varied by countries where international medical graduates were trained, after restricting to eight countries with the largest number of international medical graduates going to the US (India, Pakistan, Philippines, Syria, Nigeria, Mexico, Egypt, and China) to avoid unstable estimates (see supplementary appendix 1 for more details). Patient involvement No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are no plans to disseminate the results of the research to study participants or the relevant patient community.

Results Hospital, physician, and patient characteristics of international graduates Overall, 44.3% (19 589/44 227) of general internists in the US were international medical graduates. Among the internists, international graduates tended to be younger than US graduates (46.1 v 47.9 years, P<0.001; table 1⇓) and were more likely to work in medium sized, non-teaching for profit hospitals, and hospitals without intensive care units. Table 1 Physician and hospital characteristics of general internists who graduated from a medical school outside the US compared with graduates of a US medical school. Values are numbers (percentages) unless stated otherwise View this table: In general, compared with patients treated by US graduates, those treated by international graduates were more likely to be non-white, have lower median household income, have Medicaid coverage, and have more comorbid conditions, including congestive heart failure, chronic obstructive pulmonary disease, and diabetes (table 2⇓). Table 2 Characteristics of patients treated by general internists who graduated from a medical school outside the US compared with graduates of a US medical school. Values are numbers (percentages) unless stated otherwise View this table: Mortality Our final sample for mortality analysis included 1 215 490 patients who were treated by 44 227 physicians. The overall 30 day mortality was 11.4% (138 519/1 215 490); unadjusted mortality was 11.0% for international graduates (72 269/657 565) compared with 11.9% for US graduates (66 250/557 925). Patients treated by the international graduates had lower risk adjusted mortality compared with those treated by US graduates (adjusted mortality 11.1% v 11.7%; adjusted odds ratio 0.94, 95% confidence interval 0.92 to 0.95; P<0.001; table 3⇓). This relationship remained essentially unchanged after additional adjustment for physician characteristics (adjusted mortality 11.2% v 11.6%; adjusted odds ratio 0.96, 0.94 to 0.98; P<0.001), or after further adjustment for hospital fixed effects, which effectively compared the international graduates with US graduates within the same hospital (adjusted mortality 11.2% v 11.6%; adjusted odds ratio 0.95, 0.93 to 0.96; P<0.001). Table 3 Patient 30 day mortality between general internists who graduated from a medical school outside the US compared with graduates of a US medical school. Adjusted patient outcomes are percentages View this table: Readmission rates and costs of care Our final sample, for the analysis of readmission, comprised of 1 182 268 patients who were treated by 44 201 physicians. The overall 30 day readmission rate was 15.4% (182 239/1 182 268); the unadjusted readmission rate was 16.0% for the international graduates (102 173/639 661) and 14.8% for the US graduates (80 066/542 607). Patients treated by the international graduates had higher risk adjusted readmission rates than those treated by the US graduates (15.9% v 14.9%; adjusted odds ratio 1.08, 1.07 to 1.09; P<0.001; table 4⇓). This relationship remained unaffected after additional adjustment for physician characteristics. However, when we further adjusted for hospital fixed effects, the readmission rates no longer differed between the graduates (adjusted readmission rate 15.4% v 15.5%; adjusted odds ratio 1.00, 0.98 to 1.01; P=0.54), indicating that higher unadjusted readmission rates for the international graduates were driven by differences in the hospitals in which they work (ie, international graduates tend to practice in hospitals with higher readmission rates). Table 4 Patient 30 day readmission rates and costs of care between general internists who graduated from a medical school outside the US compared with graduates of a US medical school. Adjusted patient outcomes are percentages, unless stated otherwise View this table: Our final sample, for analysis of costs, consisted of 1 276 559 patients who were treated by 44 680 physicians. Costs of care were slightly higher for the international graduates than US graduates. After adjusting for patient and physician characteristics and hospital fixed effects, the international graduates had slightly higher costs of care ($1145 v $1098; adjusted difference $47, 95% confidence interval $39 to $55; P<0.001). Patient outcomes for specific clinical conditions The international graduates had lower patient mortality than the US graduates for pneumonia and congestive heart failure. We observed a trend toward lower mortality for all conditions except urinary tract infections, but the differences were not statistically significant for some conditions (fig 1⇓ and supplementary table B). We observed no statistically significant difference in adjusted readmission rates between the US graduates and international graduates across all conditions we studied. Costs of care were slightly higher and statistically significant for the international graduates across all conditions we studied (fig 2⇓). Fig 1 Patient outcomes between international (IMGs) and US medical graduates (USMGs), by primary diagnosis. (Top) Adjusted 30 day mortality. *P<0.05 significant. **P<0.01 significant. (Bottom) Adjusted 30 day readmission rate. For all conditions P>0.05 for difference. Risk adjusted for patient and physician characteristics and hospital fixed effects. Standard errors were clustered at physician level Fig 2 Adjusted total part B costs per hospital admission between international (IMGs) and US medical graduates (USMGs). ***P<0.001 significant. Risk adjusted for patient and physician characteristics and hospital fixed effects. Standard errors were clustered at physician level Additional analyses Hospitalists who were international graduates treated patients who were similar across a broad range of characteristics compared with hospitalists who were US graduates, supporting our hypothesis that patients treated by hospitalists are quasi-randomized based on physician work schedules (see supplementary table C). Among hospitalists, the international graduates had lower patient mortality than the US graduates (adjusted mortality 10.8% v 11.4%; adjusted odds ratio 0.94, 0.91 to 0.96; P<0.001; see supplementary table D). Our overall findings among general internists (including both hospitalists and non-hospitalist general internists) were not qualitatively affected by assigning patients to physicians on the basis of evaluation and management claims; exclusion of international graduates who trained at medical schools in Central America and the Caribbean; additional adjustment for length of hospital stay, utilization of care, or discharge location; exclusion of patients with cancer or who were discharged to hospice care; and exclusion of hospitals with medical intensive care units (see supplementary tables E-G). Differences in patient outcomes between the international and US graduates were consistent across groups defined by the teaching status of hospitals (see supplementary table H). Patient outcomes varied by country where the students were trained, although our statistical power to detect differences by individual countries was limited (see supplementary figs A-C).

Discussion Using a nationally representative sample of elderly patients admitted to the hospital in the United States, we found no evidence that patient outcomes for graduates who had trained outside of the US were worse than for graduates from a US medical school. If any, patients treated by the international graduates had lower 30 day mortality than those treated by the US graduates. These differences persisted across a broad range of clinical conditions, and even among hospitalists, where patient selection might be less of a concern. We found no differences in readmission rates between patients cared for by international or US graduates, and slightly higher spending for the international graduates. Taken together, our findings should reassure policymakers and the public that our current approach to licensing international medical graduates in the US is sufficiently rigorous to ensure high quality care. There are several explanations for why international graduates might have better patient outcomes than US graduates. The current approach for allowing international medical graduates to practice in the US may select for, on average, better physicians. Indeed, the match rate for US residency programs is substantially lower for international medical graduates (49.4% ) than for US medical graduates (94.0% for graduates of US allopathic medical schools),41 and therefore, it is possible that international graduates who are successful in the US matching process might represent some of the best physicians in their country of origin. The fact that the international graduates outperform the US graduates in test scores in the US lends some credence to this hypothesis.16 In addition, many of the international graduates who are currently practicing in the US likely underwent residency training twice, once in their home country and once in the US, and such intensive and prolonged training may be another reason why they might perform better. It is also possible that the international graduates might be more concerned about professional failure, and therefore are more engaged in continuous training and updating their skills and knowledge base, although we lack evidence to support or refute this hypothesis. The difference in patient mortality between the international and US graduates, an adjusted odds ratio of 0.95, is at most a modest clinical significance. Based on the risk difference of 0.4 percentage points, for every 250 patients treated by US medical graduates, one patient’s life would be saved if the quality of care were equivalent between the international graduates and US graduates. This difference in mortality is comparable to the reduction in mortality we have seen in all cause inpatient mortality between 2000 and 2010 across US hospitals,42 which is often recognized as the result of national quality improvement efforts, development of clinical guidelines, and new treatments. Strengths and limitations of this study The main strengths of this study were its use of nationally representative data, and a natural experiment study design (analysis focusing on hospitalists) to minimize the impact of unmeasured confounders. Patients treated by hospitalists are plausibly quasi-randomized based on when patients seek medical care and on the hospitalist’s work schedules. Although it is still possible that the severity of illness of patients seen by the international and US graduates might differ in unobservable characteristics, our data showed that, if any, the international graduates appear to treat patients with higher rates of chronic disease and lower socioeconomic status than do the US graduates. For example, we found that patients treated by the international graduates had lower socioeconomic status (eg, more often identified as being from racial and ethnic minorities, being from lower income neighborhoods, and having higher Medicaid coverage), which is generally associated with worse outcomes. Moreover, we are unaware of any other studies that suggest that the international graduates treat systematically healthier patients than do the US graduates. Our study has limitations. First, we could not distinguish between foreign born international medical graduates and US citizens who travel abroad for medical training. Approximately 73% of US citizens who are international medical graduates attend medical schools in Central America and the Caribbean.113839 Excluding physicians from these countries in our sensitivity analysis did not affect our findings. However, the results of our analyses by country suggest that there might be some heterogeneity among the international graduates based on where they trained and their reasons for coming to the US. Second, patient mortality may appear lower for the international graduates if coding comorbidities for their patients was more complete. While this is theoretically possible, it is unlikely that coding practices would vary substantially within the same hospital. Further, we observed lower unadjusted mortality for patients treated by the international graduates, which was not affected by coding practice. Finally, although 30 day mortality and readmission rates are widely accepted measures of quality for hospital care, they are not comprehensive. There are other aspects of inpatient care, such as patient experience, which we did not measure. Comparison with other studies The findings from our study are consistent with those from a limited set of smaller studies. A study of physicians practicing in Pennsylvania, US found that in-hospital mortality among patients with medical conditions was lower for international graduates compared with US graduates.20 However, another analysis of medical conditions from Ontario, Canada,22 and a recent study of surgeons from two US states,23 reported no difference in mortality between international graduates and locally trained doctors, suggesting that the quality of care provided by the international graduates might depend on specialties, or on the rigorousness of the licensure process which varies from country to country. Conclusions and policy implications Using national data on Medicare beneficiaries admitted to hospital, we found no evidence that patient outcomes were worse for those treated by international medical graduates than for those treated by US medical graduates. To an extent there was a difference: patients treated by the international graduates appeared to have somewhat lower mortality than patients treated by the US graduates. Our findings indicate that current standards of selecting international medical graduates for practice in the US are functioning well for at least one important dimension: inpatient outcomes. As we consider expanding our physician workforce, these results suggest that systems modeled on the current rigorous approach to incorporate international medical graduates should allow for better access to care and good outcomes. Further research is warranted to better understand whether patient outcomes differ between international medical graduates and domestically trained doctors for other specialties (eg, psychiatry, intensive care, geriatric care), in the outpatient setting, in the younger populations, and in other high income countries. What is already known on this topic Few studies have examined patient outcomes between international medical graduates and physicians trained domestically in high income countries

Small studies found mixed results as to whether patient outcomes differ between these two groups of physicians

No study has investigated differences in patient outcomes between international medical graduates and US medical graduates using nationally representative data What this study adds Among Medicare beneficiaries admitted to hospital, 30 day adjusted mortality rates were lower for international medical graduates than for US medical graduates, despite international graduates caring for patients with higher rates of chronic conditions

Current standards of selecting international medical graduates for practice in the US appear sufficiently rigorous to ensure high quality care

Footnotes Contributors: All authors conceived and designed the study, analyzed and interpreted the data, and critically revised the manuscript for important intellectual content. ABJ and AKJ acquired the data. YT and EJO carried out the statistical analysis. YT drafted the manuscript and is the guarantor.

Funding: This study received no support from any organization.

Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf. YT was supported in part by the Abe Fellowship Program (administered by Social Science Research Council with funds from the Japan Foundation Center for Global Partnership) and St Luke’s International University. ABJ was supported by the Office of the Director, National Institutes of Health (NIH early independence award, grant 1DP5OD017897-01). ABJ reports receiving consulting fees unrelated to this work from Pfizer, Hill Rom Services, Bristol Myers Squibb, Novartis Pharmaceuticals, Vertex Pharmaceuticals, and Precision Health Economics (a company providing consulting services to the life sciences industry).

Ethical approval: This study was reviewed and granted exemption by the institutional review board of Harvard Medical School.

Data sharing: No additional data available.

Transparency: The lead author (YT) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.