Hostname: page-component-7c8c6479df-8mjnm Total loading time: 0 Render date: 2024-03-28T08:54:18.669Z Has data issue: false hasContentIssue false

Do regulated resident working hours affect medical graduate education? Trends in the American psychiatry board pass rates pre- and post-2003 duty hours regulations

Published online by Cambridge University Press:  02 January 2018

Gaurav Jain
Affiliation:
Southern Illinois University School of Medicine, Springfield, USA
Kristina Dzara
Affiliation:
Yale University School of Medicine, New Haven, USA
Mir Nadeem Mazhar*
Affiliation:
Queen's University, Kingston, Ontario, Canada
Manisha Punwani
Affiliation:
University of California San Francisco, USA
*
Mir Nadeem Mazhar (mnm1@queensu.ca)
Rights & Permissions [Opens in a new window]

Abstract

Aims and method

To assess trends of the American Board of Psychiatry and Neurology examination pass rates before and after the 2003 duty hours regulations (DHR). We obtained the pass rates for part I and II for years 2000–2010. Data were divided pre-DHR (2000–2003) and post-DHR (2007–2010).

Results

During the pre-DHR period, first- and multiple-attempt group pass rates were 80.7% and 39.0% which changed in the post-DHR period to 89.7% and 39.1% respectively. Similarly for the part II exam, the pre-DHR first- and multiple-attempt group pass rates were 60.2% and 43.5% respectively, which increased to 78.7% and 53.8%, among the post-DHR group. Overall, there was a significant increase in the first-attempt candidates pass rates for parts I and II, whereas multiple-attempt candidates did not benefit as strongly.

Clinical implications

The results suggest that the 2003 DHR may have had a positive impact on examination-based medical knowledge in psychiatry.

Type
Education & Training
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © Royal College of Psychiatrists, 2014

In July 2003, the US Accreditation Council for Graduate Medical Education (ACGME) initially mandated nationwide duty hours regulations (DHR) on resident work hours; this lasted until June 2011, when additional rule changes were implemented. Reference Philibert, Friedmann and Williams1,Reference Nasca, Day and Amis2 While the aim of these changes was to improve working conditions and patient safety, the medical profession has raised concerns about potential negative effects on graduate medical education (GME). Reference Steinbrook3Reference West, Cook, Popkave and Kolars5 Most published studies were subjective and single-institution survey-based, with small sample sizes. Reference Fletcher, Reed and Arora6Reference Moonesinghe, Lowery, Shahi, Millen and Beard8 Only a few utilised objective measures (such as operative case-loads, standardised in-training examination scores, or the board pass rates at a single or multiple institutions) to examine the impact of the 2003 DHR on the quality of GME and all of them were from surgical specialties. Reference Fletcher, Reed and Arora6Reference Moonesinghe, Lowery, Shahi, Millen and Beard8 The literature suggests that the 2003 DHR have an influence on three main groups: trainees, faculty and patients. Reference Jamal, Rousseau, Hanna, Doi, Meterissian and Snell7,Reference Skeff, Ezeji-Okoye, Pompei and Rockson9,Reference Fletcher, Underwood III, Davis, Mangrulkar, McMahon and Saint10 Generally, trainees' perceptions about the effect of the 2003 DHR on their educational experience were neutral to somewhat positive, Reference West, Cook, Popkave and Kolars5,Reference Jagsi, Shapiro, Weissman, Dorer and Weinstein11,Reference Myers, Bellini, Morris, Graham, Katz and Potts12 contrary to training directors or faculty members who had somewhat negative to neutral opinions about the effects of DHR on trainee education. Reference West, Cook, Popkave and Kolars5,Reference Jamal, Rousseau, Hanna, Doi, Meterissian and Snell7 Meanwhile, the 2003 DHR have had varying effects on patient outcomes, depending on the indicator, however, there is some evidence of a decrease in mortality for medical and surgical patients. Reference Fletcher, Reed and Arora6,Reference Moonesinghe, Lowery, Shahi, Millen and Beard8 There is no clear consensus regarding the influence of the 2003 DHR on objective GME outcomes in the USA. Reference Fletcher, Reed and Arora6,Reference Moonesinghe, Lowery, Shahi, Millen and Beard8Reference Fletcher, Underwood III, Davis, Mangrulkar, McMahon and Saint10

The lack of evidence regarding the effect of the 2003 DHR on GME quality is even more limited in the field of psychiatry. Reference Sutton13 One published study used a debate format to report opinions of three authors. Reference Rasminsky, Lomonaco and Auchincloss14 The only national survey-based study focusing on psychiatry programmes' adherence to the 2003 DHR did not assess its impact on GME. Reference Sattar, Basith, Madison and Bhatia15 Another multicentre survey-based study examined the effect of the 2011 duty hour changes on the GME in several specialties, including psychiatry, and concluded that it would have a negative impact on GME quality. Reference Drolet, Spalluto and Fischer16 All studies also noted that additional research is required to understand the impact of duty hour changes on the GME in psychiatry and other specialties. Reference Rasminsky, Lomonaco and Auchincloss14Reference Drolet, Spalluto and Fischer16

Until 2010, the American Board of Psychiatry and Neurology (ABPN) administered the psychiatry board examination in two parts. 17 Part I assessed the knowledge through a 420-item, multiple-choice examination administered via computer for 8 h. Part II (oral) assessed practical interview and clinical skills. It included the examination of simulated patients under the observation of one or more examiners. The manner of examining patients and the subsequent reasoning and deductions constituted an important part of the examination. Knowledge of basic science principles, special diagnostic procedures, management recommendations and assessment of risk were also essential aspects of the examination, which focused on evaluation of clinical skills. For 2000 through 2010, the psychiatry board certification examination was unique as it administered the oral board with a simulated patient section. 18 A major outcome measure assessing the quality and competency of trained physicians is their ‘board pass’ status. To the best of our knowledge, no study has assessed the trend of the US national board pass rate before and after the 2003 DHR and discussed the possible reasons of change, if any.

Method

We requested psychiatry board pass data (part I and part II) from the ABPN for years 2000 through 2010 matched with the candidate's age, gender, medical school location, type of residency programme (university-based, university-affiliated or community-based) and number of attempts. In 2011 a major change occurred, with the combination of both parts into a single computer-based examination. Thus, we excluded 2011 and onwards from our analyses.

However, due to certain limitations ABPN was only able to provide limited year-wise aggregate data for the psychiatry boards. For each year from 2000 through 2010, we obtained the number of total and first-attempt candidates who took the exam and the number who passed for parts I and II. To assess the effect of the 2003 DHR change, we divided the data into two groups: pre-DHR (2000–2003), before the regulations went into effect, and post-DHR (2007–2010), as 2007 was the first year when graduates fully trained under the 2003 DHR modification began to take boards. To analyse potential variability in the effect of the 2003 DHR changes by attempt type (first- v. multiple-attempt candidates), we calculated the number of candidates who attempted the test more than one time, as well as their pass rates. We defined this group as multiple-attempt candidates.

We divided the data into six subgroups for analysis: (1) part I total; (2) part I first attempt; (3) part I multiple attempt; (4) part II total; (5) part II first attempt; and (6) part II multiple attempt. We analysed the differences pre- and post-DHR change within these subgroups, using SPSS on Windows. Chi-squared tests determined whether there were significant differences between the pre- and post-DHR subgroups. Finally, for each of the six groups, we calculated odds ratios to determine the likelihood of those in the post-DHR group passing the boards, compared with those who took boards prior to the DHR change. We also calculated 95% confidence intervals.

Results

Table 1 displays the number of total and first-attempt candidates, number passed and pass rates of the ABPN's psychiatry part I and II examinations, over time, for years 2000–2010. The calculated multiple-attempt candidates pass rates are also displayed. There is a trend of a gradual increase in the total pass rates and first-attempt pass rates for both part I and part II. Multiple-attempt pass rates were generally stable for part I, whereas there is an upward trend for part II.

For part I, the average total, first-attempt and multiple-attempt pass rates before the 2003 DHR change (2000–2003) were 64.0%, 80.7% and 39.0% respectively. Post-DHR change (2007–2010), these increased to 76.8%, 89.7% and 39.1%. Similarly for part II, 53.5%, 60.2% and 43.5% of the total, first-attempt and multiple-attempt candidates passed pre-DHR respectively. Post-DHR change, these rates increased to 71.8%, 78.7% and 53.8%.

Table 2 presents the results of the analyses comparing total, first-attempt and multiple-attempt pass rates between the pre- and post-DHR change groups. For part I, compared with those who took the tests pre-DHR changes, odds ratios suggested a total increase of 86.5% (95% CI 1.734–2.005) in the likelihood of passing. First-attempt candidates increased their odds of passing by 109.3% (95% CI 1.864–2.350). However, multiple-attempt candidates did not have a significant increase (P = 0.945).

Post-DHR change, the overall odds of passing part II increased by 120.3% (95% CI 2.056–2.360). First-attempt candidates increased their odds by 143.5% (95% CI 2.226–2.663), and multiple-attempt candidates' odds increased by 51.4% (95% CI 1.351–1.697).

Discussion

Although a large amount of discourse surrounds the effect of DHR changes on the quality of GME, few studies have utilised objective measures to evaluate its impact. Three review articles noted that ten studies assessed the impact of the 2003 DHR changes on standardised examination scores: Reference Fletcher, Reed and Arora6,Reference Moonesinghe, Lowery, Shahi, Millen and Beard8 nine from surgical specialties and one from obstetrics and gynaecology. Two reported an improvement in test scores after the DHR, and two reported a decrease; five showed no change in scores. Only one study compared American Board of Surgery pass rates pre- and post-2003 DHR, although comparisons were limited to 17 sites in New England. Reference Sneider, Larkin and Shah19 Our study found improved pass rates in the post-2003 DHR cohort, compared with the pre-DHR cohort, as measured through national psychiatry board pass rates in the USA. In addition, first-time test takers, who likely recently graduated, appear to have improved performance, which may in part be due to these changes. This finding is supported by the marked increase in first-attempt candidates' pass rates for part I, compared with the multiple-attempt candidates who appear to receive little benefit from the DHR changes. This is unique and suggests that DHR changes may have helped residents improve their examination-based medical knowledge.

Table 1 American Board of Psychiatry and Neurology pass rates for part I and part II examinations (2000-2010)Footnote a

Candidates
n
Passed
n (%)
FAC
n
FAC passed
n (%)
MAC
passed
n (%)Footnote b
Part I
    2010 1840 1415 (76.9) 1412 1253 (88.7) 37.9
    2009 1795 1421 (79.2) 1400 1264 (90.3) 39.7
    2008 1575 1229 (78.0) 1139 1057 (92.8) 39.4
    2007 1734 1269 (73.2) 1222 1068 (87.4) 39.3
    2006 1826 1329 (72.8) 1236 1056 (85.4) 46.3
    2005 1816 1281 (70.5) 1196 996 (83.3) 46.0
    2004 1680 1137 (67.7) 1119 900 (80.4) 42.2
    2003 1815 1282 (70.6) 1186 1002 (84.5) 44.5
    2002 1812 1203 (66.4) 1098 896 (81.6) 43.0
    2001 1960 1176 (60.0) 1149 912 (79.4) 32.6
    2000 2105 1261 (59.9) 1180 912 (77.3) 37.7
Part II
    2010 1761 1307 (74.2) 1319 1043 (79.1) 59.7
    2009 1815 1311 (72.2) 1347 1070 (79.4) 51.5
    2008 1657 1192 (71.9) 1172 931 (79.4) 53.8
    2007 1750 1203 (68.7) 1208 926 (76.7) 51.1
    2006 1766 1199 (67.9) 1148 870 (75.8) 53.2
    2005 1806 1129 (62.5) 991 705 (71.1) 52.0
    2004 2146 1157 (53.9) 1260 789 (62.6) 41.5
    2003 1996 1041 (52.2) 1104 650 (58.9) 43.8
    2002 1881 967 (51.4) 1138 660 (58.0) 41.3
    2001 1867 997 (53.4) 1123 689 (61.4) 41.4
    2000 1922 1097 (57.1) 1213 759 (62.6) 47.7

FAC, first-attempt candidates; MAC, multiple-attempt candidates.

b. Multiple attempt candidates include those who have taken the test more than once.

MAC pass rate = ( total passed FAC passed ) 100 ( total candidates FAC )

Limitations

There are several limitations of our study. First, there were significant changes in the examination style and evaluation. Reference Faulkner20 In April 2005, the part II scoring system changed and in May 2006 the part II test format changed for the audiovisual (non-patient) section. Further, in 2008 the part I test administration date changed from fall to summer. Each of these may have had an independent effect on the pass rates.

Second, in July 2001 the ACGME introduced the teaching and assessment of core competencies as a part of

Table 2 Pass rate comparison between pre- and post-2003 duty hour restriction periods

Part I pass rates Part II pass rates
Total FAC MACFootnote a Total FAC MACFootnote a
2000-2003 64.0% 80.7% 39.0% 53.5% 60.2% 43.5%
2007-2010 76.8% 89.7% 39.1% 71.8% 78.7% 53.8%
P Footnote *** Footnote *** P = 0.945 Footnote *** Footnote *** Footnote ***
Odds ratio 1.865 2.093 1.004 2.203 2.435 1.514
(95% CI) (1.734-2.005) (1.864-2.350) (0.891-1.132) (2.056-2.360) (2.226-2.663) (1.351-1.697)

FAC, first-attempt candidates; MAC, multiple-attempt candidates.

a. Multiple-attempt candidates include those who have taken the test more than once.

*** P<0.0001.

common programme requirements. Reference Batalden, Leach, Swing, Dreyfus and Dreyfus21 The ACGME is emphasising phase-in and progressive improvement in the implementation of these competencies in four phases; Reference Batalden, Leach, Swing, Dreyfus and Dreyfus21 in July 2011, the implementation entered into its final phase. This progressive change may have influenced pass rates. Overall, these two changes were made to streamline the process of training and testing, hence they would have likely contributed to the positive change. Specifically, the changes to the psychiatry part II examination may have had a positive impact on pass rates. However, our results suggested that there was a stronger impact on the likelihood of passing part II among the first-attempt candidates than multiple-attempt candidates. Therefore, the difference between pre- and post-DHR part II pass rates cannot be accounted for solely by the examination system changes.

Third, because of data limitations, we were unable to compute full statistical models controlling for all variables which may have had an effect on pass rates. These variables include but are not limited to age, gender, medical school location, United States Medical Licensing Examination scores, type of residency programme (university-based, university-affiliated or community-based), prior training, and utilisation of board preparation courses.

Despite these limitations, the total and first-attempt pass rates suggest an overall positive trend, which may be partly attributed to the DHR changes. Future studies to eliminate these limitations and including a new post-2011 DHR group may be helpful in ascertaining the contribution of the DHR changes in a candidate's board examination result.

This is the first study which analysed national board pass rates in an attempt to assess the impact of the 2003 DHR changes on the examination-based medical knowledge of board-eligible candidates. Overall, we found a significant trend of increases in the total and first-attempt pass rates for both part I and part II of the ABPN psychiatry boards. However, due to limitations we were unable to determine the amount of contribution made by the DHR change in this trend. As the specialty of psychiatry has not been at the forefront of the debate over residents' work hours, our findings may not be generalisable to other specialties as well. Despite the shortcomings, this study contributes to the literature surrounding the effect of the DHR changes on GME and suggests that the changes may have made a positive contribution to this aspect of resident training in psychiatry.

Footnotes

The abstract of this study was presented as a poster at the 2012 American Psychiatric Association Annual Meeting in Philadelphia, USA. The views expressed in this paper do not reflect the official policy or position of the American Board of Psychiatry and Neurology.

Declaration of interests

None.

References

1 Philibert, I, Friedmann, P, Williams, WT. New requirements for resident duty hours. JAMA 2002; 288: 1112–4.Google Scholar
2 Nasca, TJ, Day, SH, Amis, ES. The new recommendations on duty hours from the ACGME Task Force. N Engl J Med 2010; 363: e3.CrossRefGoogle ScholarPubMed
3 Steinbrook, R. The debate over residents' work hours. N Engl J Med 2002; 347: 1296–302.Google Scholar
4 Charap, M. Reducing resident work hours: unproven assumptions and unforeseen outcomes. Ann Intern Med 2004; 140: 814–5.Google Scholar
5 West, CP, Cook, RJ, Popkave, C, Kolars, JC. Perceived impact of duty hours regulations: a survey of residents and program directors. Am J Med 2007; 120: 644–8.CrossRefGoogle ScholarPubMed
6 Fletcher, KE, Reed, DA, Arora, VM. Patient safety, resident education and resident well-being following implementation of the 2003 ACGME duty hour rules. J Gen Intern Med 2011; 26: 907–19.CrossRefGoogle ScholarPubMed
7 Jamal, MH, Rousseau, MC, Hanna, WC, Doi, SA, Meterissian, S, Snell, L. Effect of the ACGME duty hours restrictions on surgical residents and faculty: a systematic review. Acad Med 2011; 86: 3442.CrossRefGoogle ScholarPubMed
8 Moonesinghe, SR, Lowery, J, Shahi, N, Millen, A, Beard, JD. Impact of reduction in working hours for doctors in training on postgraduate medical education and patients' outcomes: systematic review. BMJ 2011; 342: d1580.Google Scholar
9 Skeff, KM, Ezeji-Okoye, S, Pompei, P, Rockson, S. Benefits of resident work hours regulation. Ann Intern Med 2004; 140: 816–7.Google Scholar
10 Fletcher, KE, Underwood III, W, Davis, SQ, Mangrulkar, RS, McMahon, LF, Saint, S. Effects of work hour reduction on residents' lives: a systematic review. JAMA 2005; 294: 1088–100.Google Scholar
11 Jagsi, R, Shapiro, J, Weissman, JS, Dorer, DJ, Weinstein, DF. The educational impact of ACGME limits on resident and fellow duty hours: a pre-post survey study. Acad Med 2006; 81: 1059–68.Google Scholar
12 Myers, JS, Bellini, LM, Morris, JB, Graham, D, Katz, J, Potts, JR, et al Internal medicine and general surgery residents' attitudes about the ACGME duty hours regulations: a multicenter study. Acad Med 2006; 81: 1052–8.Google Scholar
13 Sutton, B. The need to define and study resident duty hours. Acad Psychiatr 2004; 28: 253–5.CrossRefGoogle ScholarPubMed
14 Rasminsky, S, Lomonaco, A, Auchincloss, E. Work hours regulations for house staff in psychiatry: bad or good for residency training? Acad Psychiatr 2008; 32: 5460.Google Scholar
15 Sattar, SP, Basith, F, Madison, J, Bhatia, SC. New ACGME work-hour guidelines and their impact on current residency training practices. Acad Psychiatr 2005; 29: 279–82.CrossRefGoogle ScholarPubMed
16 Drolet, BC, Spalluto, LB,Fischer, SA. Residents' perspectives on ACGME regulation of supervision and duty hours – a national survey. N Engl J Med 2010; 363: e34.Google Scholar
17 American Board of Psychiatry and Neurology. 2010 Information for Applicants: Initial Certification – Psychiatry. ABPN, 2009. Available at http://www.abpn.com/downloads/ifas/IFA_Cert_Psych_2010.pdf (accessed 28 Nov 2011).Google Scholar
18 American Board of Medical Specialties. 2010 ABMS Certificate Statistics: pp. 1011. ABMS, 2010.Google Scholar
19 Sneider, EB, Larkin, AC, Shah, SA. Has the 80-hour workweek improved surgical resident education in New England? J Surg Educ 2009; 66: 140–5.Google Scholar
20 Faulkner, LR. A Selective History of the ABPN: It s' Déjà Vu All Over Again (presentation). ABPN, 2009. Available at http://www.abpn.com/downloads/presentations/Selective%20History%20ABPN%20092109%20[Compatibility%20Mode].pdf (accessed 28 Nov 2011).Google Scholar
21 Batalden, P, Leach, D, Swing, S, Dreyfus, H, Dreyfus, S. General competencies and accreditation in graduate medical education. Health Affairs 2002; 21: 103–11.Google Scholar
Figure 0

Table 1 American Board of Psychiatry and Neurology pass rates for part I and part II examinations (2000-2010)a

Figure 1

Table 2 Pass rate comparison between pre- and post-2003 duty hour restriction periods

Submit a response

eLetters

No eLetters have been published for this article.