Hospitalists are increasingly responsible for educating students and housestaff in internal medicine.[1] Because the quality of teaching is an important factor in learning,[2, 3, 4] leaders in medical education have expressed concern over the rapid shift of teaching responsibilities to this new group of educators.[5, 6, 7, 8] Moreover, recent changes in duty hour restrictions have strained both student and resident education,[9, 10] necessitating the optimization of inpatient teaching.[11, 12] Many hospitalists have recently finished residency and have not had formal training in clinical teaching. Collectively, most hospital medicine groups are early in their careers, have significant clinical obligations,[13] and may not have the bandwidth or expertise to provide faculty development for improving clinical teaching.

Rationally designed and theoretically sound faculty development to improve inpatient clinical teaching is required to meet this challenge. There are a limited number of reports describing faculty development focused on strengthening the teaching of hospitalists, and only 3 utilized direct observation and feedback, 1 of which involved peer observation in the clinical setting.[14, 15, 16] This 2011 report described a narrative method of peer observation and feedback but did not assess for efficacy of the program.[16] To our knowledge, there have been no studies of structured peer observation and feedback to optimize hospitalist attendings' teaching which have evaluated the efficacy of the intervention.

We developed a faculty development program based on peer observation and feedback based on actual teaching practices, using structured feedback anchored in validated and observable measures of effective teaching. We hypothesized that participation in the program would increase confidence in key teaching skills, increase confidence in the ability to give and receive peer feedback, and strengthen attitudes toward peer observation and feedback.

METHODS

Intervention

Using a modified Delphi approach, 5 medical education experts selected the 10 items that are most easily observable and salient to formal attending teaching rounds from the SFDP‐26 teaching assessment tool. A structured observation form was created, which included a checklist of the 10 selected items, space for note taking, and a template for narrative feedback (Figure 1).

Figure 1

We introduced the SFDP framework during a 2‐hour initial training session. Participants watched videos of teaching, learned to identify the 10 selected teaching behaviors, developed appropriate constructive and reinforcing comments, and practiced giving and receiving peer feedback.

Dyads were created on the basis of predetermined attending schedules. Participants were asked to observe and be observed twice during attending teaching rounds over the course of the academic year. Attending teaching rounds were defined as any preplanned didactic activity for ward teams. The structured observation forms were returned to the study coordinators after the observer had given feedback to the presenter. A copy of the feedback without the observer's notes was also given to each speaker. At the midpoint of the academic year, a refresher session was offered to reinforce those teaching behaviors that were the least frequently performed to date. All participants received a $50.00 Amazon.com gift card, and additional gift card incentives were offered to the dyads that first completed both observations.

RESULTS

DISCUSSION

Previous studies have shown that teaching skills are unlikely to improve without feedback,[28, 29, 30] yet feedback for hospitalists is usually limited to summative, end‐rotation evaluations from learners, disconnected from the teaching encounter. Our theory‐based, rationally designed peer observation and feedback program resulted in increased confidence in the ability to give feedback, receive feedback, and teach effectively. Participation did not result in negative attitudes toward giving and receiving feedback from colleagues. Participants self‐reported increased performance of important teaching behaviors. Most participants rated the program very highly, and endorsed improved teaching skills as a result of the program.

Our experience provides several lessons for other groups considering the implementation of peer feedback to strengthen teaching. First, we suggest that hospitalist groups may expect variable degrees of participation in a voluntary peer feedback program. In our program, 41% of eligible attendings did not participate. We did not specifically investigate why; we speculate that they may not have had the time, believed that their teaching skills were already strong, or they may have been daunted at the idea of peer review. It is also possible that participants were a self‐selected group who were the most motivated to strengthen their teaching. Second, we note the steep decline in the number of observations in the second half of the year. Informal assessment for reasons for the drop‐off suggested that after initial enthusiasm for the program, navigating the logistics of observing the same peer in the second half of the year proved to be prohibitive to many participants. Therefore, future versions of peer feedback programs may benefit from removing the dyad requirement and encouraging all participants to observe one another whenever possible.

With these lessons in mind, we believe that a peer observation program could be implemented by other hospital medicine groups. The program does not require extensive content expertise or senior faculty but does require engaged leadership and interested and motivated faculty. Groups could identify an individual in their group with an interest in clinical teaching who could then be responsible for creating the training session (materials available upon request). We believe that with only a small upfront investment, most hospital medicine groups could use this as a model to build a peer observation program aimed at improving clinical teaching.

Our study has several limitations. As noted above, our participation rate was 59%, and the number of participating attendings declined through the year. We did not examine whether our program resulted in advances in the knowledge, skills, or attitudes of the learners; because each attending teaching session was unique, it was not possible to measure changes in learner knowledge. Our primary outcome measures relied on self‐assessment rather than higher order and more objective measures of teaching efficacy. Furthermore, our results may not be generalizable to other programs, given the heterogeneity in service structures and teaching practices across the country. This was an uncontrolled study; some of the outcomes may have naturally occurred independent of the intervention due to the natural evolution of clinical teaching. As with any educational intervention that integrates multiple strategies, we are not able to discern if the improved outcomes were the result of the initial didactic sessions, the refresher sessions, or the peer feedback itself. Serial assessments of frequency of teaching behaviors were not done due to the low number of observations in the second half of the program. Finally, our 10‐item tool derived from the validated SFDP‐26 tool is not itself a validated assessment of teaching.

We acknowledge that the increased confidence seen in our participants does not necessarily predict improved performance. Although increased confidence in core skills is a necessary step that can lead to changes in behavior, further studies are needed to determine whether the increase in faculty confidence that results from peer observation and feedback translates into improved educational outcomes.

The pressure on hospitalists to be excellent teachers is here to stay. Resources to train these faculty are scarce, yet we must prioritize faculty development in teaching to optimize the training of future physicians. Our data illustrate the benefits of peer observation and feedback. Hospitalist programs should consider this option in addressing the professional development needs of their faculty.

Hospitalists are increasingly responsible for educating students and housestaff in internal medicine.[1] Because the quality of teaching is an important factor in learning,[2, 3, 4] leaders in medical education have expressed concern over the rapid shift of teaching responsibilities to this new group of educators.[5, 6, 7, 8] Moreover, recent changes in duty hour restrictions have strained both student and resident education,[9, 10] necessitating the optimization of inpatient teaching.[11, 12] Many hospitalists have recently finished residency and have not had formal training in clinical teaching. Collectively, most hospital medicine groups are early in their careers, have significant clinical obligations,[13] and may not have the bandwidth or expertise to provide faculty development for improving clinical teaching.

Rationally designed and theoretically sound faculty development to improve inpatient clinical teaching is required to meet this challenge. There are a limited number of reports describing faculty development focused on strengthening the teaching of hospitalists, and only 3 utilized direct observation and feedback, 1 of which involved peer observation in the clinical setting.[14, 15, 16] This 2011 report described a narrative method of peer observation and feedback but did not assess for efficacy of the program.[16] To our knowledge, there have been no studies of structured peer observation and feedback to optimize hospitalist attendings' teaching which have evaluated the efficacy of the intervention.

We developed a faculty development program based on peer observation and feedback based on actual teaching practices, using structured feedback anchored in validated and observable measures of effective teaching. We hypothesized that participation in the program would increase confidence in key teaching skills, increase confidence in the ability to give and receive peer feedback, and strengthen attitudes toward peer observation and feedback.

METHODS

Intervention

Using a modified Delphi approach, 5 medical education experts selected the 10 items that are most easily observable and salient to formal attending teaching rounds from the SFDP‐26 teaching assessment tool. A structured observation form was created, which included a checklist of the 10 selected items, space for note taking, and a template for narrative feedback (Figure 1).

Figure 1

We introduced the SFDP framework during a 2‐hour initial training session. Participants watched videos of teaching, learned to identify the 10 selected teaching behaviors, developed appropriate constructive and reinforcing comments, and practiced giving and receiving peer feedback.

Dyads were created on the basis of predetermined attending schedules. Participants were asked to observe and be observed twice during attending teaching rounds over the course of the academic year. Attending teaching rounds were defined as any preplanned didactic activity for ward teams. The structured observation forms were returned to the study coordinators after the observer had given feedback to the presenter. A copy of the feedback without the observer's notes was also given to each speaker. At the midpoint of the academic year, a refresher session was offered to reinforce those teaching behaviors that were the least frequently performed to date. All participants received a $50.00 Amazon.com gift card, and additional gift card incentives were offered to the dyads that first completed both observations.

RESULTS

DISCUSSION

Previous studies have shown that teaching skills are unlikely to improve without feedback,[28, 29, 30] yet feedback for hospitalists is usually limited to summative, end‐rotation evaluations from learners, disconnected from the teaching encounter. Our theory‐based, rationally designed peer observation and feedback program resulted in increased confidence in the ability to give feedback, receive feedback, and teach effectively. Participation did not result in negative attitudes toward giving and receiving feedback from colleagues. Participants self‐reported increased performance of important teaching behaviors. Most participants rated the program very highly, and endorsed improved teaching skills as a result of the program.

Our experience provides several lessons for other groups considering the implementation of peer feedback to strengthen teaching. First, we suggest that hospitalist groups may expect variable degrees of participation in a voluntary peer feedback program. In our program, 41% of eligible attendings did not participate. We did not specifically investigate why; we speculate that they may not have had the time, believed that their teaching skills were already strong, or they may have been daunted at the idea of peer review. It is also possible that participants were a self‐selected group who were the most motivated to strengthen their teaching. Second, we note the steep decline in the number of observations in the second half of the year. Informal assessment for reasons for the drop‐off suggested that after initial enthusiasm for the program, navigating the logistics of observing the same peer in the second half of the year proved to be prohibitive to many participants. Therefore, future versions of peer feedback programs may benefit from removing the dyad requirement and encouraging all participants to observe one another whenever possible.

With these lessons in mind, we believe that a peer observation program could be implemented by other hospital medicine groups. The program does not require extensive content expertise or senior faculty but does require engaged leadership and interested and motivated faculty. Groups could identify an individual in their group with an interest in clinical teaching who could then be responsible for creating the training session (materials available upon request). We believe that with only a small upfront investment, most hospital medicine groups could use this as a model to build a peer observation program aimed at improving clinical teaching.

Our study has several limitations. As noted above, our participation rate was 59%, and the number of participating attendings declined through the year. We did not examine whether our program resulted in advances in the knowledge, skills, or attitudes of the learners; because each attending teaching session was unique, it was not possible to measure changes in learner knowledge. Our primary outcome measures relied on self‐assessment rather than higher order and more objective measures of teaching efficacy. Furthermore, our results may not be generalizable to other programs, given the heterogeneity in service structures and teaching practices across the country. This was an uncontrolled study; some of the outcomes may have naturally occurred independent of the intervention due to the natural evolution of clinical teaching. As with any educational intervention that integrates multiple strategies, we are not able to discern if the improved outcomes were the result of the initial didactic sessions, the refresher sessions, or the peer feedback itself. Serial assessments of frequency of teaching behaviors were not done due to the low number of observations in the second half of the program. Finally, our 10‐item tool derived from the validated SFDP‐26 tool is not itself a validated assessment of teaching.

We acknowledge that the increased confidence seen in our participants does not necessarily predict improved performance. Although increased confidence in core skills is a necessary step that can lead to changes in behavior, further studies are needed to determine whether the increase in faculty confidence that results from peer observation and feedback translates into improved educational outcomes.

The pressure on hospitalists to be excellent teachers is here to stay. Resources to train these faculty are scarce, yet we must prioritize faculty development in teaching to optimize the training of future physicians. Our data illustrate the benefits of peer observation and feedback. Hospitalist programs should consider this option in addressing the professional development needs of their faculty.

Hospitalists are increasingly responsible for educating students and housestaff in internal medicine.[1] Because the quality of teaching is an important factor in learning,[2, 3, 4] leaders in medical education have expressed concern over the rapid shift of teaching responsibilities to this new group of educators.[5, 6, 7, 8] Moreover, recent changes in duty hour restrictions have strained both student and resident education,[9, 10] necessitating the optimization of inpatient teaching.[11, 12] Many hospitalists have recently finished residency and have not had formal training in clinical teaching. Collectively, most hospital medicine groups are early in their careers, have significant clinical obligations,[13] and may not have the bandwidth or expertise to provide faculty development for improving clinical teaching.

Rationally designed and theoretically sound faculty development to improve inpatient clinical teaching is required to meet this challenge. There are a limited number of reports describing faculty development focused on strengthening the teaching of hospitalists, and only 3 utilized direct observation and feedback, 1 of which involved peer observation in the clinical setting.[14, 15, 16] This 2011 report described a narrative method of peer observation and feedback but did not assess for efficacy of the program.[16] To our knowledge, there have been no studies of structured peer observation and feedback to optimize hospitalist attendings' teaching which have evaluated the efficacy of the intervention.

We developed a faculty development program based on peer observation and feedback based on actual teaching practices, using structured feedback anchored in validated and observable measures of effective teaching. We hypothesized that participation in the program would increase confidence in key teaching skills, increase confidence in the ability to give and receive peer feedback, and strengthen attitudes toward peer observation and feedback.

METHODS

Intervention

Using a modified Delphi approach, 5 medical education experts selected the 10 items that are most easily observable and salient to formal attending teaching rounds from the SFDP‐26 teaching assessment tool. A structured observation form was created, which included a checklist of the 10 selected items, space for note taking, and a template for narrative feedback (Figure 1).

Figure 1

We introduced the SFDP framework during a 2‐hour initial training session. Participants watched videos of teaching, learned to identify the 10 selected teaching behaviors, developed appropriate constructive and reinforcing comments, and practiced giving and receiving peer feedback.

Dyads were created on the basis of predetermined attending schedules. Participants were asked to observe and be observed twice during attending teaching rounds over the course of the academic year. Attending teaching rounds were defined as any preplanned didactic activity for ward teams. The structured observation forms were returned to the study coordinators after the observer had given feedback to the presenter. A copy of the feedback without the observer's notes was also given to each speaker. At the midpoint of the academic year, a refresher session was offered to reinforce those teaching behaviors that were the least frequently performed to date. All participants received a $50.00 Amazon.com gift card, and additional gift card incentives were offered to the dyads that first completed both observations.

RESULTS

DISCUSSION

Previous studies have shown that teaching skills are unlikely to improve without feedback,[28, 29, 30] yet feedback for hospitalists is usually limited to summative, end‐rotation evaluations from learners, disconnected from the teaching encounter. Our theory‐based, rationally designed peer observation and feedback program resulted in increased confidence in the ability to give feedback, receive feedback, and teach effectively. Participation did not result in negative attitudes toward giving and receiving feedback from colleagues. Participants self‐reported increased performance of important teaching behaviors. Most participants rated the program very highly, and endorsed improved teaching skills as a result of the program.

Our experience provides several lessons for other groups considering the implementation of peer feedback to strengthen teaching. First, we suggest that hospitalist groups may expect variable degrees of participation in a voluntary peer feedback program. In our program, 41% of eligible attendings did not participate. We did not specifically investigate why; we speculate that they may not have had the time, believed that their teaching skills were already strong, or they may have been daunted at the idea of peer review. It is also possible that participants were a self‐selected group who were the most motivated to strengthen their teaching. Second, we note the steep decline in the number of observations in the second half of the year. Informal assessment for reasons for the drop‐off suggested that after initial enthusiasm for the program, navigating the logistics of observing the same peer in the second half of the year proved to be prohibitive to many participants. Therefore, future versions of peer feedback programs may benefit from removing the dyad requirement and encouraging all participants to observe one another whenever possible.

With these lessons in mind, we believe that a peer observation program could be implemented by other hospital medicine groups. The program does not require extensive content expertise or senior faculty but does require engaged leadership and interested and motivated faculty. Groups could identify an individual in their group with an interest in clinical teaching who could then be responsible for creating the training session (materials available upon request). We believe that with only a small upfront investment, most hospital medicine groups could use this as a model to build a peer observation program aimed at improving clinical teaching.

Our study has several limitations. As noted above, our participation rate was 59%, and the number of participating attendings declined through the year. We did not examine whether our program resulted in advances in the knowledge, skills, or attitudes of the learners; because each attending teaching session was unique, it was not possible to measure changes in learner knowledge. Our primary outcome measures relied on self‐assessment rather than higher order and more objective measures of teaching efficacy. Furthermore, our results may not be generalizable to other programs, given the heterogeneity in service structures and teaching practices across the country. This was an uncontrolled study; some of the outcomes may have naturally occurred independent of the intervention due to the natural evolution of clinical teaching. As with any educational intervention that integrates multiple strategies, we are not able to discern if the improved outcomes were the result of the initial didactic sessions, the refresher sessions, or the peer feedback itself. Serial assessments of frequency of teaching behaviors were not done due to the low number of observations in the second half of the program. Finally, our 10‐item tool derived from the validated SFDP‐26 tool is not itself a validated assessment of teaching.

We acknowledge that the increased confidence seen in our participants does not necessarily predict improved performance. Although increased confidence in core skills is a necessary step that can lead to changes in behavior, further studies are needed to determine whether the increase in faculty confidence that results from peer observation and feedback translates into improved educational outcomes.

The pressure on hospitalists to be excellent teachers is here to stay. Resources to train these faculty are scarce, yet we must prioritize faculty development in teaching to optimize the training of future physicians. Our data illustrate the benefits of peer observation and feedback. Hospitalist programs should consider this option in addressing the professional development needs of their faculty.

Late afternoon hospital discharges are thought to create admission bottlenecks in the emergency department (ED).[1] ED overcrowding increases the length of stay (LOS) of patients[2] and is a major dissatisfier for both patients and staff.[3] In our medical center, ED patients who are admitted after 1:00 _pm have a 0.6‐day longer risk‐adjusted LOS than those admitted before 1:00 _pm (M. Radford, MD, written communication, March 2012).

Many potential barriers to discharging patients early in the day exist.[4] However, comprehensive discharge planning favorably impacts discharge times.[5] There are limited published data regarding discharging patients early in the day. Studies have focused on improved discharge care coordination,[6, 7] in‐room display of planned discharge time,[8] and a discharge brunch.[9] In January 2012, the calendar month discharge before noon (DBN) percentage for 2 inpatient medicine units in our institution was approximately 7%, well below the organizational goal of 30%. We describe an intervention to sustainably increase the DBN percentage.

METHODS

RESULTS

Primary Outcome: Calendar Month DBN Percentage

The calendar month DBN percentage increased in the first month of the intervention, from 16% to 42% (Figure 1). This improvement was sustained throughout the intervention, with an average calendar month DBN percentage of 38% over the 13‐month intervention period. Use of a 2‐tailed z test to compare the pre‐intervention proportion (11%) of patients who were DBN with the post‐intervention proportion (38%) who were DBN showed a statistically significant change (z score 23.6, P = 0.0002). Units 17E and 17W had a combined 2536 total discharges in the baseline period, with 265 patients discharged before noon. In the intervention period, 3277 total discharges occurred, with 1236 patients discharged before noon. The average time of discharge moved 1 hour and 31 minutes, from 3:43 _pm in the baseline period to 2:13 _pm in the intervention period.

Figure 1

Secondary Outcomes: O/E LOS and 30‐Day Readmission Rate

The average O/E LOS during the baseline period was 1.06, and this declined during the intervention period to 0.96 [Table 2]. Using the Wilcoxon rank sum test, we found a statistically significant difference between the O/E LOS in the baseline (median 0.82) and intervention (median 0.76) periods (P = 0.0001). The average 30‐day readmission rate declined from 14.3% during the baseline to 13.1% during the intervention period. The change in 30‐day readmission rate was not statistically significant (z score=1.3132, P=0.1902). The change in readmission rate was similar and not statistically significant whether the patient was discharged before (13.6% baseline vs 12.6% intervention, P=0.66) or after noon (14.4% baseline vs 13.4% intervention, P=0.35) (Figure 2).

Table 2.O/E LOS and 30‐Day Readmission Rate in the Baseline and Intervention Period for Units 17E and 17W

Units 17E and 17W	Baseline Period	Intervention Period	Change	P Value
NOTE: Abbreviations: O/E LOS, observed‐to‐expected length of stay. a O/E LOS statistical comparison for this non‐normally distributed continuous variable was done with the Wilcoxon rank sum test of the median value in the 2 time periods.
O/E LOS, mean (median)	1.06 (0.82)	0.96 (0.76)	10%	0.0001a
30‐day readmission rate	14.3	13.1	1.2%	0.1902

Figure 2

DISCUSSION

Throughput and discharges late in the day are challenges that face all medical centers. We demonstrate that successful and sustainable improvements in DBN are possible. We were able to increase the DBN percentage from 11% in the baseline period to an average of 38% in the 13 months after our intervention. Our success allowed us to surpass our medical center's initial goal of 30% DBN.

The intervention took place on 2 inpatient medical units at an urban, academic medical center. This intervention is likely generalizable to comparable medical centers. The study is limited by the implementation of multiple interventions as part of the DBN initiative at the same time. We are unable to isolate the effect of individual changes. If other medical centers wish to use a similar intervention, we believe the 3 most important parts of our intervention are: (1) kickoff event to engage all staff with a clear definition of roles; (2) daily real‐time feedback, utilizing tools such as unit boards tracking the DBN percentage; and (3) a standardized form of communication for expected DBNs. In our experience, for a DBN to be successful, the team, patient, and family members must be alerted, and discharge plans must be initiated at least 1 day prior to the expected discharge. Attempting to discharge a patient before noon when they have been identified on the day of discharge is a losing proposition, both to achieve a coordinated, safe discharge and for staff and patient satisfaction.

The O/E LOS and 30‐day readmission rate declined over the intervention period, suggesting that there is no negative effect on these metrics. There was concern that staff would choose to keep patients an extra night to allow for an extra DBN the following day. This was actively discouraged during the kickoff event and throughout the intervention period at interdisciplinary rounds and through informal communications. Based upon the decline in O/E LOS, this did not occur. There was also concern that the 30‐day readmission rate may increase if patients are discharged earlier in the day than usual. We observed an actual but not statistically significant decline in 30‐day readmission rate, potentially due to improved communication between team members and earlier identification of expected discharges at the prior day's afternoon DBN rounds. It is unknown if the decline in O/E LOS and 30‐day readmission rate was effected by the DBN initiative. Many other initiatives were ongoing within the medical center that could be effecting these variables. More research is required to better understand the true effect of DBN on LOS and 30‐day readmission rate.

There is limited literature on discharge early in the day. One previous study showed improvement in the DBN percentage on an obstetric floor through the institution of a discharge brunch.[9] Another report showed a modest increase (from 19.6% to 26%) in DBNs with the use of scheduled discharges.[6] This study was of unclear duration and was not specific to medical units. Another study focused on the use of in‐room display boards to document the expected day and time of patient discharge.[8] That report focused on the ability to schedule and achieve the scheduled discharge date and time. The authors describe a trend toward more discharges early in the day but provide no specific data on this effect. The only study looking specifically at discharge early in the day on a medical unit showed improvement in the discharge before 1:00 _pm percentage, but was of small size (81 total patients) and short duration (1 month).[7] Our study is of larger size and longer duration, is focused on implementation of a medical service, and provides a comprehensive system that should be reproducible.

There are several next steps to our work. We will continue to monitor the DBN percentage and the ongoing sustainability of the project. We plan to investigate the effect of this rapid and notable increase in DBN percentage on a variety of patient outcomes and hospital metrics, including patient satisfaction, timeliness of ED admissions, intensive care unit transfers to the medical floor, and direct admissions.

Our study demonstrates that increased timely discharge is an achievable and sustainable goal for medical centers. Future work will allow for better understanding of the full effects of such an intervention on patient outcomes and hospital metrics.

Late afternoon hospital discharges are thought to create admission bottlenecks in the emergency department (ED).[1] ED overcrowding increases the length of stay (LOS) of patients[2] and is a major dissatisfier for both patients and staff.[3] In our medical center, ED patients who are admitted after 1:00 _pm have a 0.6‐day longer risk‐adjusted LOS than those admitted before 1:00 _pm (M. Radford, MD, written communication, March 2012).

Many potential barriers to discharging patients early in the day exist.[4] However, comprehensive discharge planning favorably impacts discharge times.[5] There are limited published data regarding discharging patients early in the day. Studies have focused on improved discharge care coordination,[6, 7] in‐room display of planned discharge time,[8] and a discharge brunch.[9] In January 2012, the calendar month discharge before noon (DBN) percentage for 2 inpatient medicine units in our institution was approximately 7%, well below the organizational goal of 30%. We describe an intervention to sustainably increase the DBN percentage.

METHODS

RESULTS

Primary Outcome: Calendar Month DBN Percentage

The calendar month DBN percentage increased in the first month of the intervention, from 16% to 42% (Figure 1). This improvement was sustained throughout the intervention, with an average calendar month DBN percentage of 38% over the 13‐month intervention period. Use of a 2‐tailed z test to compare the pre‐intervention proportion (11%) of patients who were DBN with the post‐intervention proportion (38%) who were DBN showed a statistically significant change (z score 23.6, P = 0.0002). Units 17E and 17W had a combined 2536 total discharges in the baseline period, with 265 patients discharged before noon. In the intervention period, 3277 total discharges occurred, with 1236 patients discharged before noon. The average time of discharge moved 1 hour and 31 minutes, from 3:43 _pm in the baseline period to 2:13 _pm in the intervention period.

Figure 1

Secondary Outcomes: O/E LOS and 30‐Day Readmission Rate

The average O/E LOS during the baseline period was 1.06, and this declined during the intervention period to 0.96 [Table 2]. Using the Wilcoxon rank sum test, we found a statistically significant difference between the O/E LOS in the baseline (median 0.82) and intervention (median 0.76) periods (P = 0.0001). The average 30‐day readmission rate declined from 14.3% during the baseline to 13.1% during the intervention period. The change in 30‐day readmission rate was not statistically significant (z score=1.3132, P=0.1902). The change in readmission rate was similar and not statistically significant whether the patient was discharged before (13.6% baseline vs 12.6% intervention, P=0.66) or after noon (14.4% baseline vs 13.4% intervention, P=0.35) (Figure 2).

Table 2.O/E LOS and 30‐Day Readmission Rate in the Baseline and Intervention Period for Units 17E and 17W

Units 17E and 17W	Baseline Period	Intervention Period	Change	P Value
NOTE: Abbreviations: O/E LOS, observed‐to‐expected length of stay. a O/E LOS statistical comparison for this non‐normally distributed continuous variable was done with the Wilcoxon rank sum test of the median value in the 2 time periods.
O/E LOS, mean (median)	1.06 (0.82)	0.96 (0.76)	10%	0.0001a
30‐day readmission rate	14.3	13.1	1.2%	0.1902

Figure 2

DISCUSSION

Throughput and discharges late in the day are challenges that face all medical centers. We demonstrate that successful and sustainable improvements in DBN are possible. We were able to increase the DBN percentage from 11% in the baseline period to an average of 38% in the 13 months after our intervention. Our success allowed us to surpass our medical center's initial goal of 30% DBN.

The intervention took place on 2 inpatient medical units at an urban, academic medical center. This intervention is likely generalizable to comparable medical centers. The study is limited by the implementation of multiple interventions as part of the DBN initiative at the same time. We are unable to isolate the effect of individual changes. If other medical centers wish to use a similar intervention, we believe the 3 most important parts of our intervention are: (1) kickoff event to engage all staff with a clear definition of roles; (2) daily real‐time feedback, utilizing tools such as unit boards tracking the DBN percentage; and (3) a standardized form of communication for expected DBNs. In our experience, for a DBN to be successful, the team, patient, and family members must be alerted, and discharge plans must be initiated at least 1 day prior to the expected discharge. Attempting to discharge a patient before noon when they have been identified on the day of discharge is a losing proposition, both to achieve a coordinated, safe discharge and for staff and patient satisfaction.

The O/E LOS and 30‐day readmission rate declined over the intervention period, suggesting that there is no negative effect on these metrics. There was concern that staff would choose to keep patients an extra night to allow for an extra DBN the following day. This was actively discouraged during the kickoff event and throughout the intervention period at interdisciplinary rounds and through informal communications. Based upon the decline in O/E LOS, this did not occur. There was also concern that the 30‐day readmission rate may increase if patients are discharged earlier in the day than usual. We observed an actual but not statistically significant decline in 30‐day readmission rate, potentially due to improved communication between team members and earlier identification of expected discharges at the prior day's afternoon DBN rounds. It is unknown if the decline in O/E LOS and 30‐day readmission rate was effected by the DBN initiative. Many other initiatives were ongoing within the medical center that could be effecting these variables. More research is required to better understand the true effect of DBN on LOS and 30‐day readmission rate.

There is limited literature on discharge early in the day. One previous study showed improvement in the DBN percentage on an obstetric floor through the institution of a discharge brunch.[9] Another report showed a modest increase (from 19.6% to 26%) in DBNs with the use of scheduled discharges.[6] This study was of unclear duration and was not specific to medical units. Another study focused on the use of in‐room display boards to document the expected day and time of patient discharge.[8] That report focused on the ability to schedule and achieve the scheduled discharge date and time. The authors describe a trend toward more discharges early in the day but provide no specific data on this effect. The only study looking specifically at discharge early in the day on a medical unit showed improvement in the discharge before 1:00 _pm percentage, but was of small size (81 total patients) and short duration (1 month).[7] Our study is of larger size and longer duration, is focused on implementation of a medical service, and provides a comprehensive system that should be reproducible.

There are several next steps to our work. We will continue to monitor the DBN percentage and the ongoing sustainability of the project. We plan to investigate the effect of this rapid and notable increase in DBN percentage on a variety of patient outcomes and hospital metrics, including patient satisfaction, timeliness of ED admissions, intensive care unit transfers to the medical floor, and direct admissions.

Our study demonstrates that increased timely discharge is an achievable and sustainable goal for medical centers. Future work will allow for better understanding of the full effects of such an intervention on patient outcomes and hospital metrics.

Late afternoon hospital discharges are thought to create admission bottlenecks in the emergency department (ED).[1] ED overcrowding increases the length of stay (LOS) of patients[2] and is a major dissatisfier for both patients and staff.[3] In our medical center, ED patients who are admitted after 1:00 _pm have a 0.6‐day longer risk‐adjusted LOS than those admitted before 1:00 _pm (M. Radford, MD, written communication, March 2012).

Many potential barriers to discharging patients early in the day exist.[4] However, comprehensive discharge planning favorably impacts discharge times.[5] There are limited published data regarding discharging patients early in the day. Studies have focused on improved discharge care coordination,[6, 7] in‐room display of planned discharge time,[8] and a discharge brunch.[9] In January 2012, the calendar month discharge before noon (DBN) percentage for 2 inpatient medicine units in our institution was approximately 7%, well below the organizational goal of 30%. We describe an intervention to sustainably increase the DBN percentage.

METHODS

RESULTS

Primary Outcome: Calendar Month DBN Percentage

The calendar month DBN percentage increased in the first month of the intervention, from 16% to 42% (Figure 1). This improvement was sustained throughout the intervention, with an average calendar month DBN percentage of 38% over the 13‐month intervention period. Use of a 2‐tailed z test to compare the pre‐intervention proportion (11%) of patients who were DBN with the post‐intervention proportion (38%) who were DBN showed a statistically significant change (z score 23.6, P = 0.0002). Units 17E and 17W had a combined 2536 total discharges in the baseline period, with 265 patients discharged before noon. In the intervention period, 3277 total discharges occurred, with 1236 patients discharged before noon. The average time of discharge moved 1 hour and 31 minutes, from 3:43 _pm in the baseline period to 2:13 _pm in the intervention period.

Figure 1

Secondary Outcomes: O/E LOS and 30‐Day Readmission Rate

The average O/E LOS during the baseline period was 1.06, and this declined during the intervention period to 0.96 [Table 2]. Using the Wilcoxon rank sum test, we found a statistically significant difference between the O/E LOS in the baseline (median 0.82) and intervention (median 0.76) periods (P = 0.0001). The average 30‐day readmission rate declined from 14.3% during the baseline to 13.1% during the intervention period. The change in 30‐day readmission rate was not statistically significant (z score=1.3132, P=0.1902). The change in readmission rate was similar and not statistically significant whether the patient was discharged before (13.6% baseline vs 12.6% intervention, P=0.66) or after noon (14.4% baseline vs 13.4% intervention, P=0.35) (Figure 2).

Table 2.O/E LOS and 30‐Day Readmission Rate in the Baseline and Intervention Period for Units 17E and 17W

Units 17E and 17W	Baseline Period	Intervention Period	Change	P Value
NOTE: Abbreviations: O/E LOS, observed‐to‐expected length of stay. a O/E LOS statistical comparison for this non‐normally distributed continuous variable was done with the Wilcoxon rank sum test of the median value in the 2 time periods.
O/E LOS, mean (median)	1.06 (0.82)	0.96 (0.76)	10%	0.0001a
30‐day readmission rate	14.3	13.1	1.2%	0.1902

Figure 2

DISCUSSION

Throughput and discharges late in the day are challenges that face all medical centers. We demonstrate that successful and sustainable improvements in DBN are possible. We were able to increase the DBN percentage from 11% in the baseline period to an average of 38% in the 13 months after our intervention. Our success allowed us to surpass our medical center's initial goal of 30% DBN.

The intervention took place on 2 inpatient medical units at an urban, academic medical center. This intervention is likely generalizable to comparable medical centers. The study is limited by the implementation of multiple interventions as part of the DBN initiative at the same time. We are unable to isolate the effect of individual changes. If other medical centers wish to use a similar intervention, we believe the 3 most important parts of our intervention are: (1) kickoff event to engage all staff with a clear definition of roles; (2) daily real‐time feedback, utilizing tools such as unit boards tracking the DBN percentage; and (3) a standardized form of communication for expected DBNs. In our experience, for a DBN to be successful, the team, patient, and family members must be alerted, and discharge plans must be initiated at least 1 day prior to the expected discharge. Attempting to discharge a patient before noon when they have been identified on the day of discharge is a losing proposition, both to achieve a coordinated, safe discharge and for staff and patient satisfaction.

The O/E LOS and 30‐day readmission rate declined over the intervention period, suggesting that there is no negative effect on these metrics. There was concern that staff would choose to keep patients an extra night to allow for an extra DBN the following day. This was actively discouraged during the kickoff event and throughout the intervention period at interdisciplinary rounds and through informal communications. Based upon the decline in O/E LOS, this did not occur. There was also concern that the 30‐day readmission rate may increase if patients are discharged earlier in the day than usual. We observed an actual but not statistically significant decline in 30‐day readmission rate, potentially due to improved communication between team members and earlier identification of expected discharges at the prior day's afternoon DBN rounds. It is unknown if the decline in O/E LOS and 30‐day readmission rate was effected by the DBN initiative. Many other initiatives were ongoing within the medical center that could be effecting these variables. More research is required to better understand the true effect of DBN on LOS and 30‐day readmission rate.

There is limited literature on discharge early in the day. One previous study showed improvement in the DBN percentage on an obstetric floor through the institution of a discharge brunch.[9] Another report showed a modest increase (from 19.6% to 26%) in DBNs with the use of scheduled discharges.[6] This study was of unclear duration and was not specific to medical units. Another study focused on the use of in‐room display boards to document the expected day and time of patient discharge.[8] That report focused on the ability to schedule and achieve the scheduled discharge date and time. The authors describe a trend toward more discharges early in the day but provide no specific data on this effect. The only study looking specifically at discharge early in the day on a medical unit showed improvement in the discharge before 1:00 _pm percentage, but was of small size (81 total patients) and short duration (1 month).[7] Our study is of larger size and longer duration, is focused on implementation of a medical service, and provides a comprehensive system that should be reproducible.

There are several next steps to our work. We will continue to monitor the DBN percentage and the ongoing sustainability of the project. We plan to investigate the effect of this rapid and notable increase in DBN percentage on a variety of patient outcomes and hospital metrics, including patient satisfaction, timeliness of ED admissions, intensive care unit transfers to the medical floor, and direct admissions.

Our study demonstrates that increased timely discharge is an achievable and sustainable goal for medical centers. Future work will allow for better understanding of the full effects of such an intervention on patient outcomes and hospital metrics.

Decreased efficiency at the beginning of residency training likely results in preventable harm for patients, a phenomenon known as the July Effect.[1, 2] Postgraduate year (PGY)1 residents enter training with a variety of clinical skills and experiences, and concerns exist regarding their preparation to enter graduate medical education (GME).[3] Electrocardiogram (ECG) interpretation is a core clinical skill that residents must have on the first day of training to manage patients, recognize emergencies, and develop evidence‐based and cost‐effective treatment plans. We assessed incoming PGY‐1 residents' ability to interpret common ECG findings as part of a rigorous boot camp experience.[4]

METHODS

This was an institutional review board‐approved pre‐post study of 81 new PGY‐1 residents' ECG interpretation skills. Subjects represented all trainees from internal medicine (n=47), emergency medicine (n=13), anesthesiology (n=11), and general surgery (n=10), who entered GME at Northwestern University in June 2013. Residents completed a pretest, followed by a 60‐minute interactive small group tutorial and a post‐test. Program faculty and expert cardiologists selected 10 common ECG findings for the study, many representing medical emergencies requiring immediate treatment. The diagnoses were: normal sinus rhythm, hyperkalemia, right bundle branch block (RBBB), left bundle branch block (LBBB), complete heart block, lateral wall myocardial infarction (MI), anterior wall MI, atrial fibrillation, ventricular paced rhythm, and ventricular tachycardia (VT). ECGs were selected from an online reference set (www.ecg.bidmc.harvard.edu; Harvard University, Cambridge, MA) using different pre‐ and post‐test ECGs in varying order to represent the same 10 common findings.

RESULTS

All 81 residents completed the study. The mean age was 27 years, and 56% were male. Eighty (99%) graduated from a US medical school. The mean United States Medical Licensing Examination scores were step 1: 243.8 (14.4) and step 2: 251.8 (13.6). Twenty‐six (32%) completed a cardiology rotation in medical school. Before the pretest, residents self‐assessed their ECG interpretation skills as a mean of 61.8 (standard deviation 17.2) using a scale of 0 (not confident) to 100 (very confident). Pretest results ranged from 60.5% correct (complete heart block) to 96.3% correct (normal sinus rhythm). Eighteen residents (22%) did not recognize hyperkalemia, 20 (25%) were unable to identify RBBB, and 15 (18%) LBBB. Twenty‐two (27%) could not discern a lateral wall MI, and 8 residents (10%) missed an anterior wall MI. Sixteen (20%) could not diagnose atrial fibrillation, 18 (22%) could not identify a ventricular paced rhythm, and 13 (16%) did not recognize VT. Mean post‐test scores improved significantly for 5 cases (P<0.05), but did not rise significantly for normal sinus rhythm, lateral wall MI, anterior wall MI, hyperkalemia, and ventricular paced rhythm 1.

Figure 1

DISCUSSION

PGY‐1 residents from multiple specialties were not confident regarding their ability to interpret ECGs and could not reliably identify 10 basic findings. This is despite graduating almost exclusively from US medical schools and performing at high levels on standardized tests. Although boot camp improved recognition of important ECG findings, including VT and bundle branch blocks, identification of emergent diagnoses such as lateral/anterior MI and hyperkalemia require additional training and close supervision during patient care. This study provides further evidence that the preparation of PGY‐1 residents to enter GME is lacking. Recent calls for inclusion of cost‐consciousness and stewardship of resources as a seventh competency for residents[5] are challenging, because incoming trainees do not uniformly possess the basic clinical skills needed to make these judgments.[3, 4] If residents cannot reliably interpret ECGs, it is not possible to determine cost‐effective testing strategies for patients with cardiac conditions. Based on the result of this study and others,[3, 4] we believe medical schools should agree upon specific graduation requirements to ensure all students have mastered core competencies and are prepared to enter GME.

Decreased efficiency at the beginning of residency training likely results in preventable harm for patients, a phenomenon known as the July Effect.[1, 2] Postgraduate year (PGY)1 residents enter training with a variety of clinical skills and experiences, and concerns exist regarding their preparation to enter graduate medical education (GME).[3] Electrocardiogram (ECG) interpretation is a core clinical skill that residents must have on the first day of training to manage patients, recognize emergencies, and develop evidence‐based and cost‐effective treatment plans. We assessed incoming PGY‐1 residents' ability to interpret common ECG findings as part of a rigorous boot camp experience.[4]

METHODS

This was an institutional review board‐approved pre‐post study of 81 new PGY‐1 residents' ECG interpretation skills. Subjects represented all trainees from internal medicine (n=47), emergency medicine (n=13), anesthesiology (n=11), and general surgery (n=10), who entered GME at Northwestern University in June 2013. Residents completed a pretest, followed by a 60‐minute interactive small group tutorial and a post‐test. Program faculty and expert cardiologists selected 10 common ECG findings for the study, many representing medical emergencies requiring immediate treatment. The diagnoses were: normal sinus rhythm, hyperkalemia, right bundle branch block (RBBB), left bundle branch block (LBBB), complete heart block, lateral wall myocardial infarction (MI), anterior wall MI, atrial fibrillation, ventricular paced rhythm, and ventricular tachycardia (VT). ECGs were selected from an online reference set (www.ecg.bidmc.harvard.edu; Harvard University, Cambridge, MA) using different pre‐ and post‐test ECGs in varying order to represent the same 10 common findings.

RESULTS

All 81 residents completed the study. The mean age was 27 years, and 56% were male. Eighty (99%) graduated from a US medical school. The mean United States Medical Licensing Examination scores were step 1: 243.8 (14.4) and step 2: 251.8 (13.6). Twenty‐six (32%) completed a cardiology rotation in medical school. Before the pretest, residents self‐assessed their ECG interpretation skills as a mean of 61.8 (standard deviation 17.2) using a scale of 0 (not confident) to 100 (very confident). Pretest results ranged from 60.5% correct (complete heart block) to 96.3% correct (normal sinus rhythm). Eighteen residents (22%) did not recognize hyperkalemia, 20 (25%) were unable to identify RBBB, and 15 (18%) LBBB. Twenty‐two (27%) could not discern a lateral wall MI, and 8 residents (10%) missed an anterior wall MI. Sixteen (20%) could not diagnose atrial fibrillation, 18 (22%) could not identify a ventricular paced rhythm, and 13 (16%) did not recognize VT. Mean post‐test scores improved significantly for 5 cases (P<0.05), but did not rise significantly for normal sinus rhythm, lateral wall MI, anterior wall MI, hyperkalemia, and ventricular paced rhythm 1.

Figure 1

DISCUSSION

PGY‐1 residents from multiple specialties were not confident regarding their ability to interpret ECGs and could not reliably identify 10 basic findings. This is despite graduating almost exclusively from US medical schools and performing at high levels on standardized tests. Although boot camp improved recognition of important ECG findings, including VT and bundle branch blocks, identification of emergent diagnoses such as lateral/anterior MI and hyperkalemia require additional training and close supervision during patient care. This study provides further evidence that the preparation of PGY‐1 residents to enter GME is lacking. Recent calls for inclusion of cost‐consciousness and stewardship of resources as a seventh competency for residents[5] are challenging, because incoming trainees do not uniformly possess the basic clinical skills needed to make these judgments.[3, 4] If residents cannot reliably interpret ECGs, it is not possible to determine cost‐effective testing strategies for patients with cardiac conditions. Based on the result of this study and others,[3, 4] we believe medical schools should agree upon specific graduation requirements to ensure all students have mastered core competencies and are prepared to enter GME.

Decreased efficiency at the beginning of residency training likely results in preventable harm for patients, a phenomenon known as the July Effect.[1, 2] Postgraduate year (PGY)1 residents enter training with a variety of clinical skills and experiences, and concerns exist regarding their preparation to enter graduate medical education (GME).[3] Electrocardiogram (ECG) interpretation is a core clinical skill that residents must have on the first day of training to manage patients, recognize emergencies, and develop evidence‐based and cost‐effective treatment plans. We assessed incoming PGY‐1 residents' ability to interpret common ECG findings as part of a rigorous boot camp experience.[4]

METHODS

This was an institutional review board‐approved pre‐post study of 81 new PGY‐1 residents' ECG interpretation skills. Subjects represented all trainees from internal medicine (n=47), emergency medicine (n=13), anesthesiology (n=11), and general surgery (n=10), who entered GME at Northwestern University in June 2013. Residents completed a pretest, followed by a 60‐minute interactive small group tutorial and a post‐test. Program faculty and expert cardiologists selected 10 common ECG findings for the study, many representing medical emergencies requiring immediate treatment. The diagnoses were: normal sinus rhythm, hyperkalemia, right bundle branch block (RBBB), left bundle branch block (LBBB), complete heart block, lateral wall myocardial infarction (MI), anterior wall MI, atrial fibrillation, ventricular paced rhythm, and ventricular tachycardia (VT). ECGs were selected from an online reference set (www.ecg.bidmc.harvard.edu; Harvard University, Cambridge, MA) using different pre‐ and post‐test ECGs in varying order to represent the same 10 common findings.

RESULTS

All 81 residents completed the study. The mean age was 27 years, and 56% were male. Eighty (99%) graduated from a US medical school. The mean United States Medical Licensing Examination scores were step 1: 243.8 (14.4) and step 2: 251.8 (13.6). Twenty‐six (32%) completed a cardiology rotation in medical school. Before the pretest, residents self‐assessed their ECG interpretation skills as a mean of 61.8 (standard deviation 17.2) using a scale of 0 (not confident) to 100 (very confident). Pretest results ranged from 60.5% correct (complete heart block) to 96.3% correct (normal sinus rhythm). Eighteen residents (22%) did not recognize hyperkalemia, 20 (25%) were unable to identify RBBB, and 15 (18%) LBBB. Twenty‐two (27%) could not discern a lateral wall MI, and 8 residents (10%) missed an anterior wall MI. Sixteen (20%) could not diagnose atrial fibrillation, 18 (22%) could not identify a ventricular paced rhythm, and 13 (16%) did not recognize VT. Mean post‐test scores improved significantly for 5 cases (P<0.05), but did not rise significantly for normal sinus rhythm, lateral wall MI, anterior wall MI, hyperkalemia, and ventricular paced rhythm 1.

Figure 1

DISCUSSION

PGY‐1 residents from multiple specialties were not confident regarding their ability to interpret ECGs and could not reliably identify 10 basic findings. This is despite graduating almost exclusively from US medical schools and performing at high levels on standardized tests. Although boot camp improved recognition of important ECG findings, including VT and bundle branch blocks, identification of emergent diagnoses such as lateral/anterior MI and hyperkalemia require additional training and close supervision during patient care. This study provides further evidence that the preparation of PGY‐1 residents to enter GME is lacking. Recent calls for inclusion of cost‐consciousness and stewardship of resources as a seventh competency for residents[5] are challenging, because incoming trainees do not uniformly possess the basic clinical skills needed to make these judgments.[3, 4] If residents cannot reliably interpret ECGs, it is not possible to determine cost‐effective testing strategies for patients with cardiac conditions. Based on the result of this study and others,[3, 4] we believe medical schools should agree upon specific graduation requirements to ensure all students have mastered core competencies and are prepared to enter GME.

Determining when residents are independently prepared to perform clinical care tasks safely is not easy or understood. Educators have struggled to identify robust ways to evaluate trainees and their preparedness to treat patients while unsupervised. Trust allows the trainee to experience increasing levels of participation and responsibility in the workplace in a way that builds competence for future practice. The breadth of knowledge and skills required to become a competent and safe physician, coupled with the busy workload confound this challenge. Notably, a technically proficient trainee may not have the clinical judgment to treat patients without supervision.

The Accreditation Council of Graduate Medical Education (ACGME) has previously outlined 6 core competencies for residency training: patient care, medical knowledge, practice‐based learning and improvement, interpersonal and communication skills, professionalism, and systems‐based practice.[1] A systematic literature review suggests that traditional trainee evaluation tools are difficult to use and unreliable in measuring the competencies independently from one another, whereas certain competencies are consistently difficult to quantify in a reliable and valid way.[2] The evaluation of trainees' clinical performance despite efforts to create objective tools remain strongly influenced by subjective measures and continues to be highly variable among different evaluators.[3] Objectively measuring resident autonomy and readiness to supervise junior colleagues remains imprecise.[4]

The ACGME's Next Accreditation System (NAS) incorporates educational milestones as part of the reporting of resident training outcomes.[5] The milestones allow for the translation of the core competencies into integrative and observable abilities. Furthermore, the milestone categories are stratified into tiers to allow progress to be measured longitudinally and by task complexity using a novel assessment strategy.

The development of trust between supervisors and trainees is a critical step in decisions to allow increased responsibility and the provision of autonomous decision making, which is an important aspect of physician training. Identifying the factors that influence the supervisors' evaluation of resident competency and capability is at the crux of trainee maturation as well as patient safety.[4] Trust, defined as believability and discernment by attendings of resident physicians, plays a large role in attending evaluations of residents during their clinical rotations.[3] Trust impacts the decisions of successful performance of entrustable professional activities (EPAs), or those tasks that require mastery prior to completion of training milestones.[6] A study of entrustment decisions made by attending anesthesiologists identified the factors that contribute to the amount of autonomy given to residents, such as trainee trustworthiness, medical knowledge, and level of training.[4] The aim of our study, building on this study, was 2‐fold: (1) use deductive qualitative analysis to apply this framework to existing resident and attending data, and (2) define the categories within this framework and describe how internal medicine attending and resident physician perceptions of trust can impact clinical decision making and patient care.

METHODS

We are reporting on a secondary data analysis of interview transcripts from a study conducted on the inpatient general medicine service at the University of Chicago, an academic tertiary care medical center. The methods for data collection and full consent have been outlined previously.[7, 8, 9] The institutional review board of the University of Chicago approved this study.

Briefly, between January 2006 and November 2006, all eligible internal medicine resident physicians, postgraduate year (PGY)‐2 or PGY‐3, and attending physicians, either generalists or hospitalists, were privately interviewed within 1 week of their final call night on the inpatient general medicine rotation to assess decision making and clinical supervision during the rotation. All interviews were conducted by 1 investigator (J.F.), and discussions were audio taped and transcribed for analysis. Interviews were conducted at the conclusion of the rotation to prevent any influence on resident and attending behavior during the rotation.

The critical incident technique, a procedure used for collecting direct observations of human behavior that have critical significance on the decision‐making process, was used to solicit examples of ineffective supervision, inquiring about 2 to 3 important clinical decisions made on the most recent call night, with probes to identify issues of trust, autonomy, and decision making.[10] A critical incident can be described as one that makes a significant contribution, either positively or negatively, on the process.

Appreciative inquiry, a technique that aims to uncover the best things about the clinical encounter being explored, was used to solicit examples of effective supervision. Probes are used to identify factors, either personal or situational, that influenced the withholding or provision of resident autonomy during periods of clinical care delivery.[11]

All identifiable information was removed from the interview transcripts to protect participant and patient confidentiality. Deductive qualitative analysis was performed using the conceptual EPA framework, which describes several factors that influence the attending physicians' decisions to deem a resident trustworthy to independently fulfill a specific clinical task.[4] These factors include (1) the nature of the task, (2) the qualities of the supervisor, (3) the qualities of the trainee and the quality of the relationship between the supervisor and the trainee, and (4) the circumstances surrounding the clinical task.

The deidentified, anonymous transcripts were reviewed by 2 investigators (K.J.C., J.M.F.) and analyzed using the constant comparative methods to deductively map the content to the existing framework and generate novel sub themes.[12, 13, 14] Novel categories within each of the domains were inductively generated. Two reviewers (K.J.C., J.M.F.) independently applied the themes to a randomly selected 10% portion of the interview transcripts to assess the inter‐rater reliability. The inter‐rater agreement was assessed using the generalized kappa statistic. The discrepancies between reviewers regarding assignment of codes were resolved via discussion and third party adjudication until consensus was achieved on thematic structure. The codes were then applied to the entire dataset.

RESULTS

Between January 2006 and November 2006, 46 of 50 (88%) attending physicians and 44 of 50 (92%) resident physicians were interviewed following the conclusion of their general medicine inpatient rotation. Of attending physicians, 55% were male, 45% were female, and 38% were academic faculty hospitalists. Of the residents who completed interviews, 47% were male, 53% were female, 52% were PGY‐2, and 45% were PGY‐3.

A total of 535 mentions of trust were abstracted from the transcripts. The 4 major domains that influence trusttrainee factors (Table 1), supervisor factors (Table 2), task factors (Table 3), and systems factors (Table 4)were deductively coded with several emerging novel categories and subthemes. The domains were consistent across the postgraduate year of trainee. No differences in themes were noted, other than those explicitly stated, between the postgraduate years.

DISCUSSION

Clinical supervisors must hold the quality of care constant while balancing the amount of supervision and autonomy provided to learners in procedural tasks and clinical decision making. We found that the development of trust is multifactorial and highly contextual. It occurs under the broad constructs of task, supervisor, trainee, and environmental factors, and is well described in prior work. We also demonstrate that often what determines these broader factors is highly subjective, frequently independent of objective measures of trainee performance. Many decisions are based on personal characteristics, such as the perception of honesty, disposition, perceived confidence or perceived overconfidence of the trainee, prior experience, and expressed future field of specialty.

Our findings are consistent with prior research, but go further in describing and demonstrating the existence and innovative use of factors, other than clinical knowledge and skill, in the formation of a multidimensional construct of trust. Kennedy et al. identified 4 dimensions of trust knowledge and skill, discernment, conscientiousness, and truthfulness[15]and demonstrated that supervising physicians rely on specific processes to assess trainee trustworthiness, specifically the use of double checks and language cues. This is consistent with our results, which demonstrate that many attending physicians independently verify information, such as laboratory findings, to inform their perceptions of trainee honesty, attention to detail, and ability to follow orders reliably. Furthermore, our subthemes of communication and the demonstration of logical clinical reasoning correspond to Kennedy's use of language cues.[15] We found that language cues are used as markers of trustworthiness, particularly early on in the rotation, as a litmus test to gauge the trainee's integrity and ability to assess and treat patients unsupervised.

To date, much has been written about the importance of direct observation in the evaluation of trainees.[16, 17, 18, 19] Our results demonstrate that supervising clinicians use a multifactorial, highly nuanced, and subjective process despite validated performance‐based assessment methods, such as the objective structured clinical exam or mini‐clinical evaluation exercise, to assess competence and grant entrustement.[3] Several factors utilized to determine trustworthiness in addition to direct observation are subjective in nature, specifically the trainee's prior experience and expressed career choice.

It is encouraging that attending physicians make use of direct observations to inform decisions of entrustment, albeit in an informal and unstructured way. They also seem to take into account the context and setting in which the observation occurs, and consider both the environmental factors as well as factors that relate to the task itself.[20] For example, attendings and residents reported that team dynamics played a large role in influencing trust decisions. We also found that attending physicians rely on indirect observation and will inquire among their colleagues and other senior residents to gain information about their trainees abilities and integrity. Evaluation tools that facilitate sharing of trainees' level of preparedness, prior feedback, and experience could facilitate the determination of readiness to complete EPAs as well as the reporting of achieved milestones in accordance with the ACGME NAS.

Sharing knowledge about trainees among attendings is common and of increasing importance in the context of attending physicians' shortened exposure to trainees due to the residency work‐hour restrictions and growing productivity pressures. In our study, attending physicians described work‐hour restrictions as detrimental to trainee trustworthiness, either in the context of decreased accountability for patient care or as intrinsic to the nature of forced absences that kept trainees from fully participating in daily ward activities and knowing their patients. Attending physicians felt that trainees did not know their patients well enough to be able to make independent decisions about care. The increased transition to a shift‐based structure of inpatient medicine may result in increasingly less time for direct observation and make it more difficult for attendings to justify their decisions about engendering trust. In addition, the increased fragmentation that is noted in training secondary to the work‐hour regulations may in fact have consequences on the development of clinical skill and decision making, such that increased attention to the need for supervision and longer lead to entrustment may be needed in certain circumstances. Attendings need guidance on how to improve their ability to observe trainees in the context of the new work environment, and how to role model decision making more effectively in the compressed time exposure to housestaff.

Our study has several limitations. The organizational structure and culture of our institution are unique to 1 academic setting. This may undermine our ability to generalize these research findings and analysis to the population at large.[21] In addition, recall bias may have played into the interpretation of the interview content given the timing with which they were performed after the conclusion of the rotation. The study interviews took place in 2006, and it is reasonable to believe that some perceptions concerning duty‐hour restrictions and competency‐based graduate medical education have changed. However, from our ongoing research over the past 5 years[4] and our personal experience with entrustment factors, we believe that the participants' perceptions of trust and competency are valid and have largely remained unchanged, given the similarity in findings to the accepted ten Cate framework. In addition, this work was done following the first iteration of the work‐hour regulations but prior to the implementation of explicit supervisory levels, so it may indeed represent a truer state of the supervisory relationship before external regulations were applied. Finally, this work represents an internal medicine residency training program and may not be generalizable to other specialties that posses different cultural factors that impact the decision for entrustment. However, the congruence of our data with that of the original work of ten Cate, which was done in gynecology,[6] and that of Sterkenberg et al. in anesthesiology,[4] supports our key factors being ubiquitous to all training programs.

In conclusion, we provide new insights into subjective factors that inform the perceptions of trust and entrustment decisions by supervising physicians, specifically subjective trainee characteristics, team dynamics, and informal observation. There was agreement among attendings about which elements of competence are considered most important in their entrustment decisions related to trainee, supervisor, task, and environmental factors. Rather than undervaluing the use of personal factors in the determination of trust, we believe that acknowledgement and appreciation of these factors may be important to give supervisors more confidence and better tools to assess resident physicians, and to understand how their personality traits relate to and impact their professional competence. Our findings are relevant for the development of assessment instruments to evaluate whether medical graduates are ready for safe practice without supervision.

Determining when residents are independently prepared to perform clinical care tasks safely is not easy or understood. Educators have struggled to identify robust ways to evaluate trainees and their preparedness to treat patients while unsupervised. Trust allows the trainee to experience increasing levels of participation and responsibility in the workplace in a way that builds competence for future practice. The breadth of knowledge and skills required to become a competent and safe physician, coupled with the busy workload confound this challenge. Notably, a technically proficient trainee may not have the clinical judgment to treat patients without supervision.

The Accreditation Council of Graduate Medical Education (ACGME) has previously outlined 6 core competencies for residency training: patient care, medical knowledge, practice‐based learning and improvement, interpersonal and communication skills, professionalism, and systems‐based practice.[1] A systematic literature review suggests that traditional trainee evaluation tools are difficult to use and unreliable in measuring the competencies independently from one another, whereas certain competencies are consistently difficult to quantify in a reliable and valid way.[2] The evaluation of trainees' clinical performance despite efforts to create objective tools remain strongly influenced by subjective measures and continues to be highly variable among different evaluators.[3] Objectively measuring resident autonomy and readiness to supervise junior colleagues remains imprecise.[4]

The ACGME's Next Accreditation System (NAS) incorporates educational milestones as part of the reporting of resident training outcomes.[5] The milestones allow for the translation of the core competencies into integrative and observable abilities. Furthermore, the milestone categories are stratified into tiers to allow progress to be measured longitudinally and by task complexity using a novel assessment strategy.

The development of trust between supervisors and trainees is a critical step in decisions to allow increased responsibility and the provision of autonomous decision making, which is an important aspect of physician training. Identifying the factors that influence the supervisors' evaluation of resident competency and capability is at the crux of trainee maturation as well as patient safety.[4] Trust, defined as believability and discernment by attendings of resident physicians, plays a large role in attending evaluations of residents during their clinical rotations.[3] Trust impacts the decisions of successful performance of entrustable professional activities (EPAs), or those tasks that require mastery prior to completion of training milestones.[6] A study of entrustment decisions made by attending anesthesiologists identified the factors that contribute to the amount of autonomy given to residents, such as trainee trustworthiness, medical knowledge, and level of training.[4] The aim of our study, building on this study, was 2‐fold: (1) use deductive qualitative analysis to apply this framework to existing resident and attending data, and (2) define the categories within this framework and describe how internal medicine attending and resident physician perceptions of trust can impact clinical decision making and patient care.

METHODS

We are reporting on a secondary data analysis of interview transcripts from a study conducted on the inpatient general medicine service at the University of Chicago, an academic tertiary care medical center. The methods for data collection and full consent have been outlined previously.[7, 8, 9] The institutional review board of the University of Chicago approved this study.

Briefly, between January 2006 and November 2006, all eligible internal medicine resident physicians, postgraduate year (PGY)‐2 or PGY‐3, and attending physicians, either generalists or hospitalists, were privately interviewed within 1 week of their final call night on the inpatient general medicine rotation to assess decision making and clinical supervision during the rotation. All interviews were conducted by 1 investigator (J.F.), and discussions were audio taped and transcribed for analysis. Interviews were conducted at the conclusion of the rotation to prevent any influence on resident and attending behavior during the rotation.

The critical incident technique, a procedure used for collecting direct observations of human behavior that have critical significance on the decision‐making process, was used to solicit examples of ineffective supervision, inquiring about 2 to 3 important clinical decisions made on the most recent call night, with probes to identify issues of trust, autonomy, and decision making.[10] A critical incident can be described as one that makes a significant contribution, either positively or negatively, on the process.

Appreciative inquiry, a technique that aims to uncover the best things about the clinical encounter being explored, was used to solicit examples of effective supervision. Probes are used to identify factors, either personal or situational, that influenced the withholding or provision of resident autonomy during periods of clinical care delivery.[11]

All identifiable information was removed from the interview transcripts to protect participant and patient confidentiality. Deductive qualitative analysis was performed using the conceptual EPA framework, which describes several factors that influence the attending physicians' decisions to deem a resident trustworthy to independently fulfill a specific clinical task.[4] These factors include (1) the nature of the task, (2) the qualities of the supervisor, (3) the qualities of the trainee and the quality of the relationship between the supervisor and the trainee, and (4) the circumstances surrounding the clinical task.

The deidentified, anonymous transcripts were reviewed by 2 investigators (K.J.C., J.M.F.) and analyzed using the constant comparative methods to deductively map the content to the existing framework and generate novel sub themes.[12, 13, 14] Novel categories within each of the domains were inductively generated. Two reviewers (K.J.C., J.M.F.) independently applied the themes to a randomly selected 10% portion of the interview transcripts to assess the inter‐rater reliability. The inter‐rater agreement was assessed using the generalized kappa statistic. The discrepancies between reviewers regarding assignment of codes were resolved via discussion and third party adjudication until consensus was achieved on thematic structure. The codes were then applied to the entire dataset.

RESULTS

Between January 2006 and November 2006, 46 of 50 (88%) attending physicians and 44 of 50 (92%) resident physicians were interviewed following the conclusion of their general medicine inpatient rotation. Of attending physicians, 55% were male, 45% were female, and 38% were academic faculty hospitalists. Of the residents who completed interviews, 47% were male, 53% were female, 52% were PGY‐2, and 45% were PGY‐3.

A total of 535 mentions of trust were abstracted from the transcripts. The 4 major domains that influence trusttrainee factors (Table 1), supervisor factors (Table 2), task factors (Table 3), and systems factors (Table 4)were deductively coded with several emerging novel categories and subthemes. The domains were consistent across the postgraduate year of trainee. No differences in themes were noted, other than those explicitly stated, between the postgraduate years.

DISCUSSION

Clinical supervisors must hold the quality of care constant while balancing the amount of supervision and autonomy provided to learners in procedural tasks and clinical decision making. We found that the development of trust is multifactorial and highly contextual. It occurs under the broad constructs of task, supervisor, trainee, and environmental factors, and is well described in prior work. We also demonstrate that often what determines these broader factors is highly subjective, frequently independent of objective measures of trainee performance. Many decisions are based on personal characteristics, such as the perception of honesty, disposition, perceived confidence or perceived overconfidence of the trainee, prior experience, and expressed future field of specialty.

Our findings are consistent with prior research, but go further in describing and demonstrating the existence and innovative use of factors, other than clinical knowledge and skill, in the formation of a multidimensional construct of trust. Kennedy et al. identified 4 dimensions of trust knowledge and skill, discernment, conscientiousness, and truthfulness[15]and demonstrated that supervising physicians rely on specific processes to assess trainee trustworthiness, specifically the use of double checks and language cues. This is consistent with our results, which demonstrate that many attending physicians independently verify information, such as laboratory findings, to inform their perceptions of trainee honesty, attention to detail, and ability to follow orders reliably. Furthermore, our subthemes of communication and the demonstration of logical clinical reasoning correspond to Kennedy's use of language cues.[15] We found that language cues are used as markers of trustworthiness, particularly early on in the rotation, as a litmus test to gauge the trainee's integrity and ability to assess and treat patients unsupervised.

To date, much has been written about the importance of direct observation in the evaluation of trainees.[16, 17, 18, 19] Our results demonstrate that supervising clinicians use a multifactorial, highly nuanced, and subjective process despite validated performance‐based assessment methods, such as the objective structured clinical exam or mini‐clinical evaluation exercise, to assess competence and grant entrustement.[3] Several factors utilized to determine trustworthiness in addition to direct observation are subjective in nature, specifically the trainee's prior experience and expressed career choice.

It is encouraging that attending physicians make use of direct observations to inform decisions of entrustment, albeit in an informal and unstructured way. They also seem to take into account the context and setting in which the observation occurs, and consider both the environmental factors as well as factors that relate to the task itself.[20] For example, attendings and residents reported that team dynamics played a large role in influencing trust decisions. We also found that attending physicians rely on indirect observation and will inquire among their colleagues and other senior residents to gain information about their trainees abilities and integrity. Evaluation tools that facilitate sharing of trainees' level of preparedness, prior feedback, and experience could facilitate the determination of readiness to complete EPAs as well as the reporting of achieved milestones in accordance with the ACGME NAS.

Sharing knowledge about trainees among attendings is common and of increasing importance in the context of attending physicians' shortened exposure to trainees due to the residency work‐hour restrictions and growing productivity pressures. In our study, attending physicians described work‐hour restrictions as detrimental to trainee trustworthiness, either in the context of decreased accountability for patient care or as intrinsic to the nature of forced absences that kept trainees from fully participating in daily ward activities and knowing their patients. Attending physicians felt that trainees did not know their patients well enough to be able to make independent decisions about care. The increased transition to a shift‐based structure of inpatient medicine may result in increasingly less time for direct observation and make it more difficult for attendings to justify their decisions about engendering trust. In addition, the increased fragmentation that is noted in training secondary to the work‐hour regulations may in fact have consequences on the development of clinical skill and decision making, such that increased attention to the need for supervision and longer lead to entrustment may be needed in certain circumstances. Attendings need guidance on how to improve their ability to observe trainees in the context of the new work environment, and how to role model decision making more effectively in the compressed time exposure to housestaff.

Our study has several limitations. The organizational structure and culture of our institution are unique to 1 academic setting. This may undermine our ability to generalize these research findings and analysis to the population at large.[21] In addition, recall bias may have played into the interpretation of the interview content given the timing with which they were performed after the conclusion of the rotation. The study interviews took place in 2006, and it is reasonable to believe that some perceptions concerning duty‐hour restrictions and competency‐based graduate medical education have changed. However, from our ongoing research over the past 5 years[4] and our personal experience with entrustment factors, we believe that the participants' perceptions of trust and competency are valid and have largely remained unchanged, given the similarity in findings to the accepted ten Cate framework. In addition, this work was done following the first iteration of the work‐hour regulations but prior to the implementation of explicit supervisory levels, so it may indeed represent a truer state of the supervisory relationship before external regulations were applied. Finally, this work represents an internal medicine residency training program and may not be generalizable to other specialties that posses different cultural factors that impact the decision for entrustment. However, the congruence of our data with that of the original work of ten Cate, which was done in gynecology,[6] and that of Sterkenberg et al. in anesthesiology,[4] supports our key factors being ubiquitous to all training programs.

In conclusion, we provide new insights into subjective factors that inform the perceptions of trust and entrustment decisions by supervising physicians, specifically subjective trainee characteristics, team dynamics, and informal observation. There was agreement among attendings about which elements of competence are considered most important in their entrustment decisions related to trainee, supervisor, task, and environmental factors. Rather than undervaluing the use of personal factors in the determination of trust, we believe that acknowledgement and appreciation of these factors may be important to give supervisors more confidence and better tools to assess resident physicians, and to understand how their personality traits relate to and impact their professional competence. Our findings are relevant for the development of assessment instruments to evaluate whether medical graduates are ready for safe practice without supervision.

Determining when residents are independently prepared to perform clinical care tasks safely is not easy or understood. Educators have struggled to identify robust ways to evaluate trainees and their preparedness to treat patients while unsupervised. Trust allows the trainee to experience increasing levels of participation and responsibility in the workplace in a way that builds competence for future practice. The breadth of knowledge and skills required to become a competent and safe physician, coupled with the busy workload confound this challenge. Notably, a technically proficient trainee may not have the clinical judgment to treat patients without supervision.

The Accreditation Council of Graduate Medical Education (ACGME) has previously outlined 6 core competencies for residency training: patient care, medical knowledge, practice‐based learning and improvement, interpersonal and communication skills, professionalism, and systems‐based practice.[1] A systematic literature review suggests that traditional trainee evaluation tools are difficult to use and unreliable in measuring the competencies independently from one another, whereas certain competencies are consistently difficult to quantify in a reliable and valid way.[2] The evaluation of trainees' clinical performance despite efforts to create objective tools remain strongly influenced by subjective measures and continues to be highly variable among different evaluators.[3] Objectively measuring resident autonomy and readiness to supervise junior colleagues remains imprecise.[4]

The ACGME's Next Accreditation System (NAS) incorporates educational milestones as part of the reporting of resident training outcomes.[5] The milestones allow for the translation of the core competencies into integrative and observable abilities. Furthermore, the milestone categories are stratified into tiers to allow progress to be measured longitudinally and by task complexity using a novel assessment strategy.

The development of trust between supervisors and trainees is a critical step in decisions to allow increased responsibility and the provision of autonomous decision making, which is an important aspect of physician training. Identifying the factors that influence the supervisors' evaluation of resident competency and capability is at the crux of trainee maturation as well as patient safety.[4] Trust, defined as believability and discernment by attendings of resident physicians, plays a large role in attending evaluations of residents during their clinical rotations.[3] Trust impacts the decisions of successful performance of entrustable professional activities (EPAs), or those tasks that require mastery prior to completion of training milestones.[6] A study of entrustment decisions made by attending anesthesiologists identified the factors that contribute to the amount of autonomy given to residents, such as trainee trustworthiness, medical knowledge, and level of training.[4] The aim of our study, building on this study, was 2‐fold: (1) use deductive qualitative analysis to apply this framework to existing resident and attending data, and (2) define the categories within this framework and describe how internal medicine attending and resident physician perceptions of trust can impact clinical decision making and patient care.

METHODS

We are reporting on a secondary data analysis of interview transcripts from a study conducted on the inpatient general medicine service at the University of Chicago, an academic tertiary care medical center. The methods for data collection and full consent have been outlined previously.[7, 8, 9] The institutional review board of the University of Chicago approved this study.

Briefly, between January 2006 and November 2006, all eligible internal medicine resident physicians, postgraduate year (PGY)‐2 or PGY‐3, and attending physicians, either generalists or hospitalists, were privately interviewed within 1 week of their final call night on the inpatient general medicine rotation to assess decision making and clinical supervision during the rotation. All interviews were conducted by 1 investigator (J.F.), and discussions were audio taped and transcribed for analysis. Interviews were conducted at the conclusion of the rotation to prevent any influence on resident and attending behavior during the rotation.

The critical incident technique, a procedure used for collecting direct observations of human behavior that have critical significance on the decision‐making process, was used to solicit examples of ineffective supervision, inquiring about 2 to 3 important clinical decisions made on the most recent call night, with probes to identify issues of trust, autonomy, and decision making.[10] A critical incident can be described as one that makes a significant contribution, either positively or negatively, on the process.

Appreciative inquiry, a technique that aims to uncover the best things about the clinical encounter being explored, was used to solicit examples of effective supervision. Probes are used to identify factors, either personal or situational, that influenced the withholding or provision of resident autonomy during periods of clinical care delivery.[11]

All identifiable information was removed from the interview transcripts to protect participant and patient confidentiality. Deductive qualitative analysis was performed using the conceptual EPA framework, which describes several factors that influence the attending physicians' decisions to deem a resident trustworthy to independently fulfill a specific clinical task.[4] These factors include (1) the nature of the task, (2) the qualities of the supervisor, (3) the qualities of the trainee and the quality of the relationship between the supervisor and the trainee, and (4) the circumstances surrounding the clinical task.

The deidentified, anonymous transcripts were reviewed by 2 investigators (K.J.C., J.M.F.) and analyzed using the constant comparative methods to deductively map the content to the existing framework and generate novel sub themes.[12, 13, 14] Novel categories within each of the domains were inductively generated. Two reviewers (K.J.C., J.M.F.) independently applied the themes to a randomly selected 10% portion of the interview transcripts to assess the inter‐rater reliability. The inter‐rater agreement was assessed using the generalized kappa statistic. The discrepancies between reviewers regarding assignment of codes were resolved via discussion and third party adjudication until consensus was achieved on thematic structure. The codes were then applied to the entire dataset.

RESULTS

Between January 2006 and November 2006, 46 of 50 (88%) attending physicians and 44 of 50 (92%) resident physicians were interviewed following the conclusion of their general medicine inpatient rotation. Of attending physicians, 55% were male, 45% were female, and 38% were academic faculty hospitalists. Of the residents who completed interviews, 47% were male, 53% were female, 52% were PGY‐2, and 45% were PGY‐3.

A total of 535 mentions of trust were abstracted from the transcripts. The 4 major domains that influence trusttrainee factors (Table 1), supervisor factors (Table 2), task factors (Table 3), and systems factors (Table 4)were deductively coded with several emerging novel categories and subthemes. The domains were consistent across the postgraduate year of trainee. No differences in themes were noted, other than those explicitly stated, between the postgraduate years.

DISCUSSION

Clinical supervisors must hold the quality of care constant while balancing the amount of supervision and autonomy provided to learners in procedural tasks and clinical decision making. We found that the development of trust is multifactorial and highly contextual. It occurs under the broad constructs of task, supervisor, trainee, and environmental factors, and is well described in prior work. We also demonstrate that often what determines these broader factors is highly subjective, frequently independent of objective measures of trainee performance. Many decisions are based on personal characteristics, such as the perception of honesty, disposition, perceived confidence or perceived overconfidence of the trainee, prior experience, and expressed future field of specialty.

Our findings are consistent with prior research, but go further in describing and demonstrating the existence and innovative use of factors, other than clinical knowledge and skill, in the formation of a multidimensional construct of trust. Kennedy et al. identified 4 dimensions of trust knowledge and skill, discernment, conscientiousness, and truthfulness[15]and demonstrated that supervising physicians rely on specific processes to assess trainee trustworthiness, specifically the use of double checks and language cues. This is consistent with our results, which demonstrate that many attending physicians independently verify information, such as laboratory findings, to inform their perceptions of trainee honesty, attention to detail, and ability to follow orders reliably. Furthermore, our subthemes of communication and the demonstration of logical clinical reasoning correspond to Kennedy's use of language cues.[15] We found that language cues are used as markers of trustworthiness, particularly early on in the rotation, as a litmus test to gauge the trainee's integrity and ability to assess and treat patients unsupervised.

To date, much has been written about the importance of direct observation in the evaluation of trainees.[16, 17, 18, 19] Our results demonstrate that supervising clinicians use a multifactorial, highly nuanced, and subjective process despite validated performance‐based assessment methods, such as the objective structured clinical exam or mini‐clinical evaluation exercise, to assess competence and grant entrustement.[3] Several factors utilized to determine trustworthiness in addition to direct observation are subjective in nature, specifically the trainee's prior experience and expressed career choice.

It is encouraging that attending physicians make use of direct observations to inform decisions of entrustment, albeit in an informal and unstructured way. They also seem to take into account the context and setting in which the observation occurs, and consider both the environmental factors as well as factors that relate to the task itself.[20] For example, attendings and residents reported that team dynamics played a large role in influencing trust decisions. We also found that attending physicians rely on indirect observation and will inquire among their colleagues and other senior residents to gain information about their trainees abilities and integrity. Evaluation tools that facilitate sharing of trainees' level of preparedness, prior feedback, and experience could facilitate the determination of readiness to complete EPAs as well as the reporting of achieved milestones in accordance with the ACGME NAS.

Sharing knowledge about trainees among attendings is common and of increasing importance in the context of attending physicians' shortened exposure to trainees due to the residency work‐hour restrictions and growing productivity pressures. In our study, attending physicians described work‐hour restrictions as detrimental to trainee trustworthiness, either in the context of decreased accountability for patient care or as intrinsic to the nature of forced absences that kept trainees from fully participating in daily ward activities and knowing their patients. Attending physicians felt that trainees did not know their patients well enough to be able to make independent decisions about care. The increased transition to a shift‐based structure of inpatient medicine may result in increasingly less time for direct observation and make it more difficult for attendings to justify their decisions about engendering trust. In addition, the increased fragmentation that is noted in training secondary to the work‐hour regulations may in fact have consequences on the development of clinical skill and decision making, such that increased attention to the need for supervision and longer lead to entrustment may be needed in certain circumstances. Attendings need guidance on how to improve their ability to observe trainees in the context of the new work environment, and how to role model decision making more effectively in the compressed time exposure to housestaff.

Our study has several limitations. The organizational structure and culture of our institution are unique to 1 academic setting. This may undermine our ability to generalize these research findings and analysis to the population at large.[21] In addition, recall bias may have played into the interpretation of the interview content given the timing with which they were performed after the conclusion of the rotation. The study interviews took place in 2006, and it is reasonable to believe that some perceptions concerning duty‐hour restrictions and competency‐based graduate medical education have changed. However, from our ongoing research over the past 5 years[4] and our personal experience with entrustment factors, we believe that the participants' perceptions of trust and competency are valid and have largely remained unchanged, given the similarity in findings to the accepted ten Cate framework. In addition, this work was done following the first iteration of the work‐hour regulations but prior to the implementation of explicit supervisory levels, so it may indeed represent a truer state of the supervisory relationship before external regulations were applied. Finally, this work represents an internal medicine residency training program and may not be generalizable to other specialties that posses different cultural factors that impact the decision for entrustment. However, the congruence of our data with that of the original work of ten Cate, which was done in gynecology,[6] and that of Sterkenberg et al. in anesthesiology,[4] supports our key factors being ubiquitous to all training programs.

In conclusion, we provide new insights into subjective factors that inform the perceptions of trust and entrustment decisions by supervising physicians, specifically subjective trainee characteristics, team dynamics, and informal observation. There was agreement among attendings about which elements of competence are considered most important in their entrustment decisions related to trainee, supervisor, task, and environmental factors. Rather than undervaluing the use of personal factors in the determination of trust, we believe that acknowledgement and appreciation of these factors may be important to give supervisors more confidence and better tools to assess resident physicians, and to understand how their personality traits relate to and impact their professional competence. Our findings are relevant for the development of assessment instruments to evaluate whether medical graduates are ready for safe practice without supervision.