Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

In England, the day when trainee doctors start work for the first time in their careers or rotate to a different hospital is the first Wednesday of August. This is often referred to as the Black Wednesday in the National Health Service (NHS), as it is widely perceived that inexperience and nonfamiliarity with the new hospital systems and policies in these first few weeks lead to increased medical errors and mismanagement and may therefore cost lives.[1] However, there is very little evidence in favor of this widely held view in the NHS. A 2009 English study found a small but significant increase of 6% in the odds of death for inpatients admitted in the week following the first Wednesday in August than in the week following the last Wednesday in July, whereas a previous report did not support this.[2, 3] In the United States, the resident trainee doctor's changeover occurs in July, and its negative impact on patient outcomes is often dubbed the July phenomenon.[4] With conflicting reports of the July phenomenon on patient outcomes,[5, 6, 7] Young et al. systematically reviewed 39 studies and concluded that the July phenomenon exists in that there is increased mortality around the changeover period.[4]

It can be hypothesized that glycemic control in inpatients with diabetes would be worse in the immediate period following changeover of trainee doctors for the same reasons mentioned earlier that impact mortality. However, contrary to expectations, a recent single‐hospital study from the United States reported that changeover of resident trainee doctors did not worsen inpatient glycemic control.[8] Although the lack of confidence among trainee doctors in inpatient diabetes management has been clearly demonstrated in England,[9] the impact of August changeover of trainee doctors on inpatient glycemic control is unknown. The aim of this study was to determine whether the August changeover of trainee doctors impacted on glycemic control in inpatients with diabetes in a single English hospital.

MATERIAL AND METHODS

The study setting was a medium‐sized 550‐bed hospital in England that serves a population of approximately 360,000 residents. Capillary blood glucose (CBG) readings for adult inpatients across all wards were downloaded from the Precision Web Point‐of‐Care Data Management System (Abbott Diabetes Care Inc., Alameda, CA), an electronic database where all the CBG readings for inpatients are stored. Patient administration data were used to identify those with diabetes admitted to the hospital for at least 1 day, and only their CBG readings were included in this study. Glucometrics, a term coined by Goldberg et al., refers to standardized glucose performance metrics to assess the quality of inpatient glycemic control.[10] In this study, patient‐day glucometric measures were used, as they are considered the best indicator of inpatient glycemic control compared to other glucometrics.[10] Patient‐day glucometrics were analyzed for 4 weeks before and after Black Wednesday for the years 2012, 2013, and 2014 using Microsoft Excel 2007 (Microsoft Corp., Redmond, WA) and R version 3.1.0 (The R Foundation, Vienna, Austria). Patient‐day glucometrics analyzed were hypoglycemia (any CBG 2.2 mmol/L [40 mg/dL], any CBG 2.9 mmol/L [52 mg/dL], any CBG 3.9 mmol/L [72 mg/dL]), normoglycemia (mean CBGs between 4 and 12 mmol/L [73‐216 mg/dL]), hyperglycemia (any CBG 12.1 mmol/L [218 mg/dL]), and mean CBG. Proportions were compared using the z test, whereas sample means between the groups were compared by nonparametric Mann‐Whitney U tests, as per statistical literature.[11] All P values are 2‐tailed, and <0.05 was considered statistically significant.

Patient characteristics and healthcare professional's workload were identified as potential causes of variation in CBG readings. Regression analysis of covariance was used to identify and adjust for these factors when comparing mean glucose readings. Binomial logistic regression was used to adjust proportions of patients‐days with readings out of range and patient‐days with mean readings within range. Variables tested were length of stay as a proxy for severity of condition, number of patients whose CBG were measured in the hospital in a day as a proxy for the healthcare professional's workload, and location of the patient to account for variation in patient characteristics as the wards were specialty based. Goodness of fit was tested using the R² value in the linear model, which indicates the proportion of outcome that is explained by the model. For binomial models, McFadden's pseudo R² (pseudo‐R²_McFadden) was used as advised for logistic models. McFadden's pseudo‐R² ranges from 0 to 1, but unlike R² in ordinary linear regression, values tend to be significantly lower: McFadden's pseudo R² values between 0.2 and 0.4 indicate excellent fit.[12]

RESULTS

A total of 16,870 patient‐day CBG measures in 2730 inpatients with diabetes were analyzed. The results of all regressions are presented in Table 1. The coefficients in the first model represent the effect of each covariate on mean patient‐day CBG. For example, each extra day of hospitalization was associated with a 0.02 mmol/L (0.36 mg/dL) increase in mean patient‐day reading, ceteris paribus. The remaining models indicate the change in relative risk (in this case the proportion of patient‐days) associated with the covariates. For example, in patients who were hospitalized for 3 days, the proportion of patient‐days with at least 1 CBG greater than 12 mmol/L (216 mg/dL) was 1.01 times the comparable proportion of patients who were hospitalized for 2 days. Each additional day in the hospital significantly increased the mean CBG by 0.015 mmol/L (0.27 mg/dL) and increased the risk of having at least 1 reading below 3.9 mmol/L (72 mg/dL) or above 12 mmol/L (216 mg/dL). Monitoring more patients in a day also affected outcomes, although the effect was small. Each additional patient monitored reduced mean patient‐day CBG by 0.011 mmol/L (0.198 mg/dL) and increased the proportion of patients with at least 1 reading below 4 mmol/L (72 mg/dL) 1.01 times. Location of the patient also significantly affected CBG readings. This could have been due to either ward or patient characteristics, but lack of data on each ward's healthcare personnel and individual patient characteristics prevented further analysis of this effect, and therefore the results were used for adjustment only. All models have relatively low predictive power, as demonstrated by the low R² and pseudo‐R²_McFadden values. In the linear model that estimated the effect of covariates on mean patient‐day CBG, the R² is 0.0270, indicating that only 2.70% of results were explained by the covariates in the model. The pseudo‐R²_McFadden varied between 0.0146 and 0.0540, as presented in Table 1. Although the pseudo‐R²_McFadden generally had lower values than the R² for the linear models, values of 0.0540 and below are considered to be relatively low.[12]

Table 2 summarizes outcomes for the 3 years individually. The results suggest that all indices of inpatient glycemic control that were analyzedhypoglycemia, normoglycemia, hyperglycemia, and mean CBGdid not worsen in August compared to July that year. The results are presented after adjustment for variation in the length of stay, number of patients monitored in a day, and location of the patient. Their effect on the difference in proportions of patients with at least 1 reading out of range and mean reading within range were not statistically significant. However, their effect on mean patient‐day CBG measures was statistically significant, although the effect was only a small decrease (0.4 mmol/L or 7.2 mg/dL) in the mean CBG (see Supporting Table 1 in the online version of this article for unadjusted readings).

DISCUSSION

This study shows that contrary to expectation, inpatient glycemic control did not worsen in the 4 weeks following the August changeover of trainee doctors for the years 2012, 2013, and 2014. In fact, inpatient glycemic control was marginally better in the first 4 weeks after changeover each year compared to the preceding 4 weeks before changeover. There may be several reasons for the findings in this study. First, since 2010 in this hospital and since 2012 nationally (further to direction from NHS England Medical Director Sir Bruce Keogh), it has become established practice that newly qualified trainee doctors shadow their colleagues at work a week prior to Black Wednesday.[13, 14] The purpose of this practice, called the preparation for professional practice is to familiarize trainee doctors with the hospital protocols and systems, improve their confidence, and potentially reduce medical errors when starting work. Second, since 2012, this hospital has also implemented the Joint British Diabetes Societies' national guidelines in managing inpatients with diabetes.[15] These guidelines are widely publicized on the changeover day during the trainee doctor's induction program. Finally, since 2012, a diabetes‐specific interactive 1‐hour educational program for trainee doctors devised by this hospital was implemented during the changeover period, which takes them through practical and problem‐solving case scenarios related to inpatient glycemic management, in particular prevention of hypoglycemia and hospital‐acquired diabetic ketoacidosis.[16] Attendance was mandatory, and informal feedback from trainee doctors about the educational program was extremely positive.

There are several limitations in this study. It could be argued that trainee doctors have very little impact on glycemic control in inpatients with diabetes. In NHS hospitals, trainee doctors are often the first port of call for managing glycemic issues in inpatients both in and out of hours, who in turn may or may not call the inpatient diabetes team wherever available. Therefore, trainee doctors' impact on glycemic control in inpatients with diabetes cannot be understated. However, it is acknowledged that in this study, a number of other factors that influence inpatient glycemic control, such as individual patient characteristics, medication errors, and the knowledge and confidence levels of individual trainee doctors, were not accounted for. Nevertheless, such factors are unlikely to have been significantly different over the 3‐year period. A further limitation was the unavailability of hospital‐wide electronic CBG data prior to 2012 to determine whether changeover impacted on inpatient glycemic control prior to this period. Another limitation was the dependence on patient administration data to identify those with diabetes, as it is well recognized that coded data in hospital data management systems can be inaccurate, though this has significantly improved over the years.[17] Finally, the most important limitation is that this is a single‐hospital study, and so the results may not be applicable to other English hospitals. Nevertheless, the finding of this study is similar to the finding in the single‐hospital study from the United States.[8]

The finding that glycemic control in inpatients with diabetes did not worsen in the 4 weeks following changeover of trainee doctors compared to the 4 weeks before changeover each year suggests that appropriate forethought and planning by the deanery foundation school and the inpatient diabetes team has prevented the anticipated deterioration of glycemic control during the August changeover of trainee doctors in this English hospital.

Disclosures: R.R. and G.R. conceived and designed the study. R.R. collected data and drafted the manuscript. R.R., D.J., and G.R. analyzed and interpreted the data. D.J. provided statistical input for analysis of the data. R.R., D.J., and G.R. critically revised the manuscript for intellectual content. All authors have approved the final version. The authors report no conflicts of interest.

In England, the day when trainee doctors start work for the first time in their careers or rotate to a different hospital is the first Wednesday of August. This is often referred to as the Black Wednesday in the National Health Service (NHS), as it is widely perceived that inexperience and nonfamiliarity with the new hospital systems and policies in these first few weeks lead to increased medical errors and mismanagement and may therefore cost lives.[1] However, there is very little evidence in favor of this widely held view in the NHS. A 2009 English study found a small but significant increase of 6% in the odds of death for inpatients admitted in the week following the first Wednesday in August than in the week following the last Wednesday in July, whereas a previous report did not support this.[2, 3] In the United States, the resident trainee doctor's changeover occurs in July, and its negative impact on patient outcomes is often dubbed the July phenomenon.[4] With conflicting reports of the July phenomenon on patient outcomes,[5, 6, 7] Young et al. systematically reviewed 39 studies and concluded that the July phenomenon exists in that there is increased mortality around the changeover period.[4]

It can be hypothesized that glycemic control in inpatients with diabetes would be worse in the immediate period following changeover of trainee doctors for the same reasons mentioned earlier that impact mortality. However, contrary to expectations, a recent single‐hospital study from the United States reported that changeover of resident trainee doctors did not worsen inpatient glycemic control.[8] Although the lack of confidence among trainee doctors in inpatient diabetes management has been clearly demonstrated in England,[9] the impact of August changeover of trainee doctors on inpatient glycemic control is unknown. The aim of this study was to determine whether the August changeover of trainee doctors impacted on glycemic control in inpatients with diabetes in a single English hospital.

MATERIAL AND METHODS

The study setting was a medium‐sized 550‐bed hospital in England that serves a population of approximately 360,000 residents. Capillary blood glucose (CBG) readings for adult inpatients across all wards were downloaded from the Precision Web Point‐of‐Care Data Management System (Abbott Diabetes Care Inc., Alameda, CA), an electronic database where all the CBG readings for inpatients are stored. Patient administration data were used to identify those with diabetes admitted to the hospital for at least 1 day, and only their CBG readings were included in this study. Glucometrics, a term coined by Goldberg et al., refers to standardized glucose performance metrics to assess the quality of inpatient glycemic control.[10] In this study, patient‐day glucometric measures were used, as they are considered the best indicator of inpatient glycemic control compared to other glucometrics.[10] Patient‐day glucometrics were analyzed for 4 weeks before and after Black Wednesday for the years 2012, 2013, and 2014 using Microsoft Excel 2007 (Microsoft Corp., Redmond, WA) and R version 3.1.0 (The R Foundation, Vienna, Austria). Patient‐day glucometrics analyzed were hypoglycemia (any CBG 2.2 mmol/L [40 mg/dL], any CBG 2.9 mmol/L [52 mg/dL], any CBG 3.9 mmol/L [72 mg/dL]), normoglycemia (mean CBGs between 4 and 12 mmol/L [73‐216 mg/dL]), hyperglycemia (any CBG 12.1 mmol/L [218 mg/dL]), and mean CBG. Proportions were compared using the z test, whereas sample means between the groups were compared by nonparametric Mann‐Whitney U tests, as per statistical literature.[11] All P values are 2‐tailed, and <0.05 was considered statistically significant.

Patient characteristics and healthcare professional's workload were identified as potential causes of variation in CBG readings. Regression analysis of covariance was used to identify and adjust for these factors when comparing mean glucose readings. Binomial logistic regression was used to adjust proportions of patients‐days with readings out of range and patient‐days with mean readings within range. Variables tested were length of stay as a proxy for severity of condition, number of patients whose CBG were measured in the hospital in a day as a proxy for the healthcare professional's workload, and location of the patient to account for variation in patient characteristics as the wards were specialty based. Goodness of fit was tested using the R² value in the linear model, which indicates the proportion of outcome that is explained by the model. For binomial models, McFadden's pseudo R² (pseudo‐R²_McFadden) was used as advised for logistic models. McFadden's pseudo‐R² ranges from 0 to 1, but unlike R² in ordinary linear regression, values tend to be significantly lower: McFadden's pseudo R² values between 0.2 and 0.4 indicate excellent fit.[12]

RESULTS

A total of 16,870 patient‐day CBG measures in 2730 inpatients with diabetes were analyzed. The results of all regressions are presented in Table 1. The coefficients in the first model represent the effect of each covariate on mean patient‐day CBG. For example, each extra day of hospitalization was associated with a 0.02 mmol/L (0.36 mg/dL) increase in mean patient‐day reading, ceteris paribus. The remaining models indicate the change in relative risk (in this case the proportion of patient‐days) associated with the covariates. For example, in patients who were hospitalized for 3 days, the proportion of patient‐days with at least 1 CBG greater than 12 mmol/L (216 mg/dL) was 1.01 times the comparable proportion of patients who were hospitalized for 2 days. Each additional day in the hospital significantly increased the mean CBG by 0.015 mmol/L (0.27 mg/dL) and increased the risk of having at least 1 reading below 3.9 mmol/L (72 mg/dL) or above 12 mmol/L (216 mg/dL). Monitoring more patients in a day also affected outcomes, although the effect was small. Each additional patient monitored reduced mean patient‐day CBG by 0.011 mmol/L (0.198 mg/dL) and increased the proportion of patients with at least 1 reading below 4 mmol/L (72 mg/dL) 1.01 times. Location of the patient also significantly affected CBG readings. This could have been due to either ward or patient characteristics, but lack of data on each ward's healthcare personnel and individual patient characteristics prevented further analysis of this effect, and therefore the results were used for adjustment only. All models have relatively low predictive power, as demonstrated by the low R² and pseudo‐R²_McFadden values. In the linear model that estimated the effect of covariates on mean patient‐day CBG, the R² is 0.0270, indicating that only 2.70% of results were explained by the covariates in the model. The pseudo‐R²_McFadden varied between 0.0146 and 0.0540, as presented in Table 1. Although the pseudo‐R²_McFadden generally had lower values than the R² for the linear models, values of 0.0540 and below are considered to be relatively low.[12]

Table 2 summarizes outcomes for the 3 years individually. The results suggest that all indices of inpatient glycemic control that were analyzedhypoglycemia, normoglycemia, hyperglycemia, and mean CBGdid not worsen in August compared to July that year. The results are presented after adjustment for variation in the length of stay, number of patients monitored in a day, and location of the patient. Their effect on the difference in proportions of patients with at least 1 reading out of range and mean reading within range were not statistically significant. However, their effect on mean patient‐day CBG measures was statistically significant, although the effect was only a small decrease (0.4 mmol/L or 7.2 mg/dL) in the mean CBG (see Supporting Table 1 in the online version of this article for unadjusted readings).

DISCUSSION

This study shows that contrary to expectation, inpatient glycemic control did not worsen in the 4 weeks following the August changeover of trainee doctors for the years 2012, 2013, and 2014. In fact, inpatient glycemic control was marginally better in the first 4 weeks after changeover each year compared to the preceding 4 weeks before changeover. There may be several reasons for the findings in this study. First, since 2010 in this hospital and since 2012 nationally (further to direction from NHS England Medical Director Sir Bruce Keogh), it has become established practice that newly qualified trainee doctors shadow their colleagues at work a week prior to Black Wednesday.[13, 14] The purpose of this practice, called the preparation for professional practice is to familiarize trainee doctors with the hospital protocols and systems, improve their confidence, and potentially reduce medical errors when starting work. Second, since 2012, this hospital has also implemented the Joint British Diabetes Societies' national guidelines in managing inpatients with diabetes.[15] These guidelines are widely publicized on the changeover day during the trainee doctor's induction program. Finally, since 2012, a diabetes‐specific interactive 1‐hour educational program for trainee doctors devised by this hospital was implemented during the changeover period, which takes them through practical and problem‐solving case scenarios related to inpatient glycemic management, in particular prevention of hypoglycemia and hospital‐acquired diabetic ketoacidosis.[16] Attendance was mandatory, and informal feedback from trainee doctors about the educational program was extremely positive.

There are several limitations in this study. It could be argued that trainee doctors have very little impact on glycemic control in inpatients with diabetes. In NHS hospitals, trainee doctors are often the first port of call for managing glycemic issues in inpatients both in and out of hours, who in turn may or may not call the inpatient diabetes team wherever available. Therefore, trainee doctors' impact on glycemic control in inpatients with diabetes cannot be understated. However, it is acknowledged that in this study, a number of other factors that influence inpatient glycemic control, such as individual patient characteristics, medication errors, and the knowledge and confidence levels of individual trainee doctors, were not accounted for. Nevertheless, such factors are unlikely to have been significantly different over the 3‐year period. A further limitation was the unavailability of hospital‐wide electronic CBG data prior to 2012 to determine whether changeover impacted on inpatient glycemic control prior to this period. Another limitation was the dependence on patient administration data to identify those with diabetes, as it is well recognized that coded data in hospital data management systems can be inaccurate, though this has significantly improved over the years.[17] Finally, the most important limitation is that this is a single‐hospital study, and so the results may not be applicable to other English hospitals. Nevertheless, the finding of this study is similar to the finding in the single‐hospital study from the United States.[8]

The finding that glycemic control in inpatients with diabetes did not worsen in the 4 weeks following changeover of trainee doctors compared to the 4 weeks before changeover each year suggests that appropriate forethought and planning by the deanery foundation school and the inpatient diabetes team has prevented the anticipated deterioration of glycemic control during the August changeover of trainee doctors in this English hospital.

Disclosures: R.R. and G.R. conceived and designed the study. R.R. collected data and drafted the manuscript. R.R., D.J., and G.R. analyzed and interpreted the data. D.J. provided statistical input for analysis of the data. R.R., D.J., and G.R. critically revised the manuscript for intellectual content. All authors have approved the final version. The authors report no conflicts of interest.

In England, the day when trainee doctors start work for the first time in their careers or rotate to a different hospital is the first Wednesday of August. This is often referred to as the Black Wednesday in the National Health Service (NHS), as it is widely perceived that inexperience and nonfamiliarity with the new hospital systems and policies in these first few weeks lead to increased medical errors and mismanagement and may therefore cost lives.[1] However, there is very little evidence in favor of this widely held view in the NHS. A 2009 English study found a small but significant increase of 6% in the odds of death for inpatients admitted in the week following the first Wednesday in August than in the week following the last Wednesday in July, whereas a previous report did not support this.[2, 3] In the United States, the resident trainee doctor's changeover occurs in July, and its negative impact on patient outcomes is often dubbed the July phenomenon.[4] With conflicting reports of the July phenomenon on patient outcomes,[5, 6, 7] Young et al. systematically reviewed 39 studies and concluded that the July phenomenon exists in that there is increased mortality around the changeover period.[4]

It can be hypothesized that glycemic control in inpatients with diabetes would be worse in the immediate period following changeover of trainee doctors for the same reasons mentioned earlier that impact mortality. However, contrary to expectations, a recent single‐hospital study from the United States reported that changeover of resident trainee doctors did not worsen inpatient glycemic control.[8] Although the lack of confidence among trainee doctors in inpatient diabetes management has been clearly demonstrated in England,[9] the impact of August changeover of trainee doctors on inpatient glycemic control is unknown. The aim of this study was to determine whether the August changeover of trainee doctors impacted on glycemic control in inpatients with diabetes in a single English hospital.

MATERIAL AND METHODS

The study setting was a medium‐sized 550‐bed hospital in England that serves a population of approximately 360,000 residents. Capillary blood glucose (CBG) readings for adult inpatients across all wards were downloaded from the Precision Web Point‐of‐Care Data Management System (Abbott Diabetes Care Inc., Alameda, CA), an electronic database where all the CBG readings for inpatients are stored. Patient administration data were used to identify those with diabetes admitted to the hospital for at least 1 day, and only their CBG readings were included in this study. Glucometrics, a term coined by Goldberg et al., refers to standardized glucose performance metrics to assess the quality of inpatient glycemic control.[10] In this study, patient‐day glucometric measures were used, as they are considered the best indicator of inpatient glycemic control compared to other glucometrics.[10] Patient‐day glucometrics were analyzed for 4 weeks before and after Black Wednesday for the years 2012, 2013, and 2014 using Microsoft Excel 2007 (Microsoft Corp., Redmond, WA) and R version 3.1.0 (The R Foundation, Vienna, Austria). Patient‐day glucometrics analyzed were hypoglycemia (any CBG 2.2 mmol/L [40 mg/dL], any CBG 2.9 mmol/L [52 mg/dL], any CBG 3.9 mmol/L [72 mg/dL]), normoglycemia (mean CBGs between 4 and 12 mmol/L [73‐216 mg/dL]), hyperglycemia (any CBG 12.1 mmol/L [218 mg/dL]), and mean CBG. Proportions were compared using the z test, whereas sample means between the groups were compared by nonparametric Mann‐Whitney U tests, as per statistical literature.[11] All P values are 2‐tailed, and <0.05 was considered statistically significant.

Patient characteristics and healthcare professional's workload were identified as potential causes of variation in CBG readings. Regression analysis of covariance was used to identify and adjust for these factors when comparing mean glucose readings. Binomial logistic regression was used to adjust proportions of patients‐days with readings out of range and patient‐days with mean readings within range. Variables tested were length of stay as a proxy for severity of condition, number of patients whose CBG were measured in the hospital in a day as a proxy for the healthcare professional's workload, and location of the patient to account for variation in patient characteristics as the wards were specialty based. Goodness of fit was tested using the R² value in the linear model, which indicates the proportion of outcome that is explained by the model. For binomial models, McFadden's pseudo R² (pseudo‐R²_McFadden) was used as advised for logistic models. McFadden's pseudo‐R² ranges from 0 to 1, but unlike R² in ordinary linear regression, values tend to be significantly lower: McFadden's pseudo R² values between 0.2 and 0.4 indicate excellent fit.[12]

RESULTS

A total of 16,870 patient‐day CBG measures in 2730 inpatients with diabetes were analyzed. The results of all regressions are presented in Table 1. The coefficients in the first model represent the effect of each covariate on mean patient‐day CBG. For example, each extra day of hospitalization was associated with a 0.02 mmol/L (0.36 mg/dL) increase in mean patient‐day reading, ceteris paribus. The remaining models indicate the change in relative risk (in this case the proportion of patient‐days) associated with the covariates. For example, in patients who were hospitalized for 3 days, the proportion of patient‐days with at least 1 CBG greater than 12 mmol/L (216 mg/dL) was 1.01 times the comparable proportion of patients who were hospitalized for 2 days. Each additional day in the hospital significantly increased the mean CBG by 0.015 mmol/L (0.27 mg/dL) and increased the risk of having at least 1 reading below 3.9 mmol/L (72 mg/dL) or above 12 mmol/L (216 mg/dL). Monitoring more patients in a day also affected outcomes, although the effect was small. Each additional patient monitored reduced mean patient‐day CBG by 0.011 mmol/L (0.198 mg/dL) and increased the proportion of patients with at least 1 reading below 4 mmol/L (72 mg/dL) 1.01 times. Location of the patient also significantly affected CBG readings. This could have been due to either ward or patient characteristics, but lack of data on each ward's healthcare personnel and individual patient characteristics prevented further analysis of this effect, and therefore the results were used for adjustment only. All models have relatively low predictive power, as demonstrated by the low R² and pseudo‐R²_McFadden values. In the linear model that estimated the effect of covariates on mean patient‐day CBG, the R² is 0.0270, indicating that only 2.70% of results were explained by the covariates in the model. The pseudo‐R²_McFadden varied between 0.0146 and 0.0540, as presented in Table 1. Although the pseudo‐R²_McFadden generally had lower values than the R² for the linear models, values of 0.0540 and below are considered to be relatively low.[12]

Table 2 summarizes outcomes for the 3 years individually. The results suggest that all indices of inpatient glycemic control that were analyzedhypoglycemia, normoglycemia, hyperglycemia, and mean CBGdid not worsen in August compared to July that year. The results are presented after adjustment for variation in the length of stay, number of patients monitored in a day, and location of the patient. Their effect on the difference in proportions of patients with at least 1 reading out of range and mean reading within range were not statistically significant. However, their effect on mean patient‐day CBG measures was statistically significant, although the effect was only a small decrease (0.4 mmol/L or 7.2 mg/dL) in the mean CBG (see Supporting Table 1 in the online version of this article for unadjusted readings).

DISCUSSION

This study shows that contrary to expectation, inpatient glycemic control did not worsen in the 4 weeks following the August changeover of trainee doctors for the years 2012, 2013, and 2014. In fact, inpatient glycemic control was marginally better in the first 4 weeks after changeover each year compared to the preceding 4 weeks before changeover. There may be several reasons for the findings in this study. First, since 2010 in this hospital and since 2012 nationally (further to direction from NHS England Medical Director Sir Bruce Keogh), it has become established practice that newly qualified trainee doctors shadow their colleagues at work a week prior to Black Wednesday.[13, 14] The purpose of this practice, called the preparation for professional practice is to familiarize trainee doctors with the hospital protocols and systems, improve their confidence, and potentially reduce medical errors when starting work. Second, since 2012, this hospital has also implemented the Joint British Diabetes Societies' national guidelines in managing inpatients with diabetes.[15] These guidelines are widely publicized on the changeover day during the trainee doctor's induction program. Finally, since 2012, a diabetes‐specific interactive 1‐hour educational program for trainee doctors devised by this hospital was implemented during the changeover period, which takes them through practical and problem‐solving case scenarios related to inpatient glycemic management, in particular prevention of hypoglycemia and hospital‐acquired diabetic ketoacidosis.[16] Attendance was mandatory, and informal feedback from trainee doctors about the educational program was extremely positive.

There are several limitations in this study. It could be argued that trainee doctors have very little impact on glycemic control in inpatients with diabetes. In NHS hospitals, trainee doctors are often the first port of call for managing glycemic issues in inpatients both in and out of hours, who in turn may or may not call the inpatient diabetes team wherever available. Therefore, trainee doctors' impact on glycemic control in inpatients with diabetes cannot be understated. However, it is acknowledged that in this study, a number of other factors that influence inpatient glycemic control, such as individual patient characteristics, medication errors, and the knowledge and confidence levels of individual trainee doctors, were not accounted for. Nevertheless, such factors are unlikely to have been significantly different over the 3‐year period. A further limitation was the unavailability of hospital‐wide electronic CBG data prior to 2012 to determine whether changeover impacted on inpatient glycemic control prior to this period. Another limitation was the dependence on patient administration data to identify those with diabetes, as it is well recognized that coded data in hospital data management systems can be inaccurate, though this has significantly improved over the years.[17] Finally, the most important limitation is that this is a single‐hospital study, and so the results may not be applicable to other English hospitals. Nevertheless, the finding of this study is similar to the finding in the single‐hospital study from the United States.[8]

The finding that glycemic control in inpatients with diabetes did not worsen in the 4 weeks following changeover of trainee doctors compared to the 4 weeks before changeover each year suggests that appropriate forethought and planning by the deanery foundation school and the inpatient diabetes team has prevented the anticipated deterioration of glycemic control during the August changeover of trainee doctors in this English hospital.

Disclosures: R.R. and G.R. conceived and designed the study. R.R. collected data and drafted the manuscript. R.R., D.J., and G.R. analyzed and interpreted the data. D.J. provided statistical input for analysis of the data. R.R., D.J., and G.R. critically revised the manuscript for intellectual content. All authors have approved the final version. The authors report no conflicts of interest.

We thank Mr. Zilm and colleagues for their interest in our work.[1] Certainly, we did not intend to imply that well‐designed buildings have little value in the efficient and patient‐centered delivery of healthcare. Our main goal was to highlight (1) that patients can distinguish between facility features and actual care delivery, and poor facilities alone should not be an excuse for poor patient satisfaction; and (2) that global evaluations are more dependent on perceived quality of care than on facility features. Furthermore, we agree with many of the points raised. Certainly, patient satisfaction is but 1 measure of successful facility design, and the delivery of modern healthcare requires updated facilities. However, based on our results, we think that healthcare administrators and designers should consider the return on investment on the costly features that are incorporated purely to improve patient satisfaction rather than for safety and staff effectiveness.

Referral patterns and patient expectations are likely very different for a tertiary care hospital like ours. A different relationship between facility design and patient satisfaction may indeed exist for community hospitals. However, we would caution against making this assumption without supportive evidence. Furthermore, it is difficult to attribute lack of improvement of physician scores in our study because of a ceiling effect. The baseline scores were certainly not exemplary, and there was plenty of room for improvement.

We agree that there is a need for high‐quality research to better understand the broader impact of healthcare design on meaningful outcomes. However, we are not impressed with the quality of much of the existing research tying physical facilities with patient stress or shorter length of stay, as mentioned by Mr. Zilm and colleagues. Evidence supporting investment in expensive facilities should be evaluated with the same high standards and rigor as for other healthcare decisions.

We thank Mr. Zilm and colleagues for their interest in our work.[1] Certainly, we did not intend to imply that well‐designed buildings have little value in the efficient and patient‐centered delivery of healthcare. Our main goal was to highlight (1) that patients can distinguish between facility features and actual care delivery, and poor facilities alone should not be an excuse for poor patient satisfaction; and (2) that global evaluations are more dependent on perceived quality of care than on facility features. Furthermore, we agree with many of the points raised. Certainly, patient satisfaction is but 1 measure of successful facility design, and the delivery of modern healthcare requires updated facilities. However, based on our results, we think that healthcare administrators and designers should consider the return on investment on the costly features that are incorporated purely to improve patient satisfaction rather than for safety and staff effectiveness.

Referral patterns and patient expectations are likely very different for a tertiary care hospital like ours. A different relationship between facility design and patient satisfaction may indeed exist for community hospitals. However, we would caution against making this assumption without supportive evidence. Furthermore, it is difficult to attribute lack of improvement of physician scores in our study because of a ceiling effect. The baseline scores were certainly not exemplary, and there was plenty of room for improvement.

We agree that there is a need for high‐quality research to better understand the broader impact of healthcare design on meaningful outcomes. However, we are not impressed with the quality of much of the existing research tying physical facilities with patient stress or shorter length of stay, as mentioned by Mr. Zilm and colleagues. Evidence supporting investment in expensive facilities should be evaluated with the same high standards and rigor as for other healthcare decisions.

We thank Mr. Zilm and colleagues for their interest in our work.[1] Certainly, we did not intend to imply that well‐designed buildings have little value in the efficient and patient‐centered delivery of healthcare. Our main goal was to highlight (1) that patients can distinguish between facility features and actual care delivery, and poor facilities alone should not be an excuse for poor patient satisfaction; and (2) that global evaluations are more dependent on perceived quality of care than on facility features. Furthermore, we agree with many of the points raised. Certainly, patient satisfaction is but 1 measure of successful facility design, and the delivery of modern healthcare requires updated facilities. However, based on our results, we think that healthcare administrators and designers should consider the return on investment on the costly features that are incorporated purely to improve patient satisfaction rather than for safety and staff effectiveness.

Referral patterns and patient expectations are likely very different for a tertiary care hospital like ours. A different relationship between facility design and patient satisfaction may indeed exist for community hospitals. However, we would caution against making this assumption without supportive evidence. Furthermore, it is difficult to attribute lack of improvement of physician scores in our study because of a ceiling effect. The baseline scores were certainly not exemplary, and there was plenty of room for improvement.

We agree that there is a need for high‐quality research to better understand the broader impact of healthcare design on meaningful outcomes. However, we are not impressed with the quality of much of the existing research tying physical facilities with patient stress or shorter length of stay, as mentioned by Mr. Zilm and colleagues. Evidence supporting investment in expensive facilities should be evaluated with the same high standards and rigor as for other healthcare decisions.

We complement Dr. Siddiqui et al. on their article published in the Journal of Hospital Medicine.[1] Analysis of the role of new physical environments on care and patient satisfaction is sparse and desperately needed for this high‐cost resource in healthcare delivery. A review of the original article led us to several observations/suggestions.

The focus of the study is on perceived patient satisfaction based on 2 survey tools. As noted by the authors, there are multiple factors that must be considered related to facilitiestheir potential contribution to patient infections and falls, the ability to accommodate new technology and procedures, and the shifting practice models such as the shift from inpatient to ambulatory care. Patient‐focused care concepts are only 1 element in the design challenge and costs.

The reputation of Johns Hopkins as a major tertiary referral center is well known internationally, and it would seem reasonable to assume that many of the patients were selected or referred to the institution based on its physicians. It does not seem unreasonable to assume that facilities would play a secondary role, and that perceived satisfaction would be high regardless of the physical environment. As noted by the authors, the transferability of this finding to community hospitals and other settings is unknown.

Patient satisfaction is an important element in design, but staff satisfaction and efficiency are also significant elements in maintaining a high‐quality healthcare system. We need tools to assess the relationship between staff retention, stress levels, and medical errors and the physical environment.

The focus of the article is on the transferability of perceived satisfaction with environment to satisfaction with physician care. Previously published studies have shown a correlation with environments and views from patients rooms with reduced patient stress levels and shorter lengths of stay. Physical space should not be disregarded as a component of effective patient care.[2]

We are committed to seeking designs that are effective, safe, and adaptable to long‐term needs. We support additional research in this and other related design issues. We hope that the improvements in patient and family environments labeled as patient focused will continue to evolve to respond to real healthcare needs. It would be unfortunate if progress is diverted by misinterpretation of the articles findings.

We complement Dr. Siddiqui et al. on their article published in the Journal of Hospital Medicine.[1] Analysis of the role of new physical environments on care and patient satisfaction is sparse and desperately needed for this high‐cost resource in healthcare delivery. A review of the original article led us to several observations/suggestions.

The focus of the study is on perceived patient satisfaction based on 2 survey tools. As noted by the authors, there are multiple factors that must be considered related to facilitiestheir potential contribution to patient infections and falls, the ability to accommodate new technology and procedures, and the shifting practice models such as the shift from inpatient to ambulatory care. Patient‐focused care concepts are only 1 element in the design challenge and costs.

The reputation of Johns Hopkins as a major tertiary referral center is well known internationally, and it would seem reasonable to assume that many of the patients were selected or referred to the institution based on its physicians. It does not seem unreasonable to assume that facilities would play a secondary role, and that perceived satisfaction would be high regardless of the physical environment. As noted by the authors, the transferability of this finding to community hospitals and other settings is unknown.

Patient satisfaction is an important element in design, but staff satisfaction and efficiency are also significant elements in maintaining a high‐quality healthcare system. We need tools to assess the relationship between staff retention, stress levels, and medical errors and the physical environment.

The focus of the article is on the transferability of perceived satisfaction with environment to satisfaction with physician care. Previously published studies have shown a correlation with environments and views from patients rooms with reduced patient stress levels and shorter lengths of stay. Physical space should not be disregarded as a component of effective patient care.[2]

We are committed to seeking designs that are effective, safe, and adaptable to long‐term needs. We support additional research in this and other related design issues. We hope that the improvements in patient and family environments labeled as patient focused will continue to evolve to respond to real healthcare needs. It would be unfortunate if progress is diverted by misinterpretation of the articles findings.

We complement Dr. Siddiqui et al. on their article published in the Journal of Hospital Medicine.[1] Analysis of the role of new physical environments on care and patient satisfaction is sparse and desperately needed for this high‐cost resource in healthcare delivery. A review of the original article led us to several observations/suggestions.

The focus of the study is on perceived patient satisfaction based on 2 survey tools. As noted by the authors, there are multiple factors that must be considered related to facilitiestheir potential contribution to patient infections and falls, the ability to accommodate new technology and procedures, and the shifting practice models such as the shift from inpatient to ambulatory care. Patient‐focused care concepts are only 1 element in the design challenge and costs.

The reputation of Johns Hopkins as a major tertiary referral center is well known internationally, and it would seem reasonable to assume that many of the patients were selected or referred to the institution based on its physicians. It does not seem unreasonable to assume that facilities would play a secondary role, and that perceived satisfaction would be high regardless of the physical environment. As noted by the authors, the transferability of this finding to community hospitals and other settings is unknown.

Patient satisfaction is an important element in design, but staff satisfaction and efficiency are also significant elements in maintaining a high‐quality healthcare system. We need tools to assess the relationship between staff retention, stress levels, and medical errors and the physical environment.

The focus of the article is on the transferability of perceived satisfaction with environment to satisfaction with physician care. Previously published studies have shown a correlation with environments and views from patients rooms with reduced patient stress levels and shorter lengths of stay. Physical space should not be disregarded as a component of effective patient care.[2]

We are committed to seeking designs that are effective, safe, and adaptable to long‐term needs. We support additional research in this and other related design issues. We hope that the improvements in patient and family environments labeled as patient focused will continue to evolve to respond to real healthcare needs. It would be unfortunate if progress is diverted by misinterpretation of the articles findings.

Over the past 3 decades, emergency department observation units (EDOUs) have been increasingly implemented in the United States to supplement emergency department (ED) care in a time of increasing patient volume and hospital crowding. Given the limited availability of hospital resources, EDOUs provide emergency clinicians an extended period of time to evaluate and risk‐stratify patients without necessitating difficult‐to‐obtain outpatient follow‐up or a short‐stay hospitalization. Changes in Medicare and insurer reimbursement policies have incentivized the adoption of EDOUs, and now, over one‐third of EDs nationally offer an observation unit.[1]

Much of the observation‐science literature has been condition and institution specific, showing benefits with respect to cost, quality of care, safety, and patient satisfaction.[2, 3, 4, 5] Until now, there had not been a national study on the impact of EDOUs to investigate important outcome: hospital admission rates. Capp and colleagues, using the National Hospital Ambulatory Care Survey (NHAMCS), attempt to answer a very important question: Do EDs with observation units have lower hospital admission rates?[6] To do so, they first standardize admission rates to sociodemographic and clinical features of the patients, while adjusting for hospital‐level factors. Then they compare the risk‐standardized hospital admission rate between EDs with and without an observation unit as reported in the NHAMCS. The authors make creative and elegant use of this publicly available, national dataset to suggest that EDOUs do not decrease hospital admissions.

The authors appropriately identify some limitations of using such data to answer questions where nuanced, countervailing forces drive the outcome of interest. It is important to note the basic statistical premise that the inability to disprove the null hypothesis is not the same thing as proving that the null hypothesis is true. In other words, although this study was not able to detect a difference between admission rates for hospitals with EDOUs and those without, it cannot be absolutely taken to mean that there is no relationship. The authors clearly state that this study was underpowered given that the difference of ED risk‐standardized hospital admission rates was small and therefore is at risk of type II error. In addition, unmeasured confounding may hide a true association between EDOUs and admission rates. Both static and dynamic measures of ED volume, crowding, and boarding, as well as changes in case mix or acuity may drive adoption of EDOUs,[7] while simultaneously associated with risk of hospitalization. Without balance between the EDs with and without observation units, or longitudinal measures of EDs over time as they are implemented, we are left with potentially biased estimates.

It is also important to highlight that not all EDOUs are created equal.[8] EDs may admit patients to the observation unit based on prespecified conditions or include all comers at physician discretion. Once placed in observation status, patients may or may not be managed by specific protocols to provide guidance on timing, order, and scope of testing and decision making.

Finally, care in EDOUs may be provided by emergency physicians, hospitalists, or other clinicians such as advanced practice providers (eg, physician assistants, nurse practitioners), a distinction that likely impacts the ultimate patient disposition. In fact, the NHAMCS asks the question, What type of physicians make decisions for patients in this observation or clinical decision unit? Capp et al., however, did not include this variable to further stratify the data. Although we do not know whether or not inclusion of this factor may have ultimately changed the results, it could have implications for how distinctions in who manages EDOUs could affect admission rates.

Still, the negative findings of this study seem to raise a number of questions, which should spark a broader discussion on EDOUs. The current analysis provides an important first step toward a national understanding of EDOUs and their role in acute care. Future inquiries should account for variation in observation units and the hospitals in which they are housed as well as inclusion of meaningful outcomes beyond admission rates. A number of methodological approaches can be considered to achieve this; propensity score matching within observational data may provide better balance between facilities with and without EDOUs, whereas multicenter impact analyses using controlled before‐and‐after or cluster‐randomized trials should be considered the gold standard for studying observation unit implementation. Outcomes in these studies should include long‐term changes in health, aggregate healthcare utilization, overuse of resources that do not provide high‐value care, and impacts on how care and costs may be redistributed when patients receive more care in observation units.

Although cost containment is often touted as a cornerstone of EDOUs, it is critical to know how the costs are measured and who is paying. For example, when an option to place a patient in observation exists, might clinicians utilize it for some patients who do not require further evaluation and testing and could have been safely discharged?[9] This observation creep may arise because clinicians can use EDOUs, not because they should. Motivations may include delaying difficult disposition decisions, avoiding uncertainty or liability when discharging patients, limited access to outpatient follow‐up, or a desire to utilize observation status to justify the existence of EDOUs within the institution. In this way, EDOUs may, in fact, provide low‐value care at a time of soaring healthcare costs.

Perhaps even more perplexing is the question of how costs are shifted through use of EDOUs.[10, 11] Much of the literature advertising its cost savings are only from the perspective of the insurers' or hospitals' perspective,[12] with 1 study estimating a potential annual cost savings of $4.6 million for each hospital, or $3 billion nationally, associated with the implementation of observation care.[5] But are medical centers just passing costs on to patients to avoid penalties and disincentives associated with short‐stay hospitalizations? Both private insurers and the Centers for Medicare and Medicaid Services may deny payments for admissions deemed unnecessary. Further, under the Affordable Care Act, avoiding hospitalizations may mean fewer penalties when Medicare patients later require admission for certain conditions. As such, hospitals may find huge incentives and cost savings associated with observation units. However, using EDOUs to avoid the Medicare readmission penalty may backfire when less‐sick patients requiring care beyond the ED are treated and discharged from observation, leaving more medically complex and ill patients for hospitalization, a group potentially more likely to be rehospitalized within 30 days, making readmission rates appear higher.

Nonetheless, because services provided during observation status are billed as an outpatient visit, patients may be liable for a proportion of the overall visit. In contrast to inpatient stays where, in general, patients owe a single copay for most or all of services rendered, outpatient visits typically involve a la carte billing. When accounting for costs related to professional and facilities fees, medications, laboratory tests, and advanced diagnostics and procedures, patient bills may be markedly higher when they are placed in observation status. This is especially true for patients covered by Medicare, where observation stays are not covered under Part A.

Research will need to simultaneously identify best practices for how EDOUs are implemented and administered while appraising their impact on patient‐centered outcomes and true costs, from multiple perspectives, including the patient, hospital, and healthcare system. There is reason to be optimistic about EDOUs as potentially high‐value components of the acute care delivery system. However, the widespread implementation of observation units with the assumption that it is cost saving to hospitals and insurers, without high‐quality population studies to inform their impact more broadly, may undermine acceptance by patients and health‐policy experts.

Over the past 3 decades, emergency department observation units (EDOUs) have been increasingly implemented in the United States to supplement emergency department (ED) care in a time of increasing patient volume and hospital crowding. Given the limited availability of hospital resources, EDOUs provide emergency clinicians an extended period of time to evaluate and risk‐stratify patients without necessitating difficult‐to‐obtain outpatient follow‐up or a short‐stay hospitalization. Changes in Medicare and insurer reimbursement policies have incentivized the adoption of EDOUs, and now, over one‐third of EDs nationally offer an observation unit.[1]

Much of the observation‐science literature has been condition and institution specific, showing benefits with respect to cost, quality of care, safety, and patient satisfaction.[2, 3, 4, 5] Until now, there had not been a national study on the impact of EDOUs to investigate important outcome: hospital admission rates. Capp and colleagues, using the National Hospital Ambulatory Care Survey (NHAMCS), attempt to answer a very important question: Do EDs with observation units have lower hospital admission rates?[6] To do so, they first standardize admission rates to sociodemographic and clinical features of the patients, while adjusting for hospital‐level factors. Then they compare the risk‐standardized hospital admission rate between EDs with and without an observation unit as reported in the NHAMCS. The authors make creative and elegant use of this publicly available, national dataset to suggest that EDOUs do not decrease hospital admissions.

The authors appropriately identify some limitations of using such data to answer questions where nuanced, countervailing forces drive the outcome of interest. It is important to note the basic statistical premise that the inability to disprove the null hypothesis is not the same thing as proving that the null hypothesis is true. In other words, although this study was not able to detect a difference between admission rates for hospitals with EDOUs and those without, it cannot be absolutely taken to mean that there is no relationship. The authors clearly state that this study was underpowered given that the difference of ED risk‐standardized hospital admission rates was small and therefore is at risk of type II error. In addition, unmeasured confounding may hide a true association between EDOUs and admission rates. Both static and dynamic measures of ED volume, crowding, and boarding, as well as changes in case mix or acuity may drive adoption of EDOUs,[7] while simultaneously associated with risk of hospitalization. Without balance between the EDs with and without observation units, or longitudinal measures of EDs over time as they are implemented, we are left with potentially biased estimates.

It is also important to highlight that not all EDOUs are created equal.[8] EDs may admit patients to the observation unit based on prespecified conditions or include all comers at physician discretion. Once placed in observation status, patients may or may not be managed by specific protocols to provide guidance on timing, order, and scope of testing and decision making.

Finally, care in EDOUs may be provided by emergency physicians, hospitalists, or other clinicians such as advanced practice providers (eg, physician assistants, nurse practitioners), a distinction that likely impacts the ultimate patient disposition. In fact, the NHAMCS asks the question, What type of physicians make decisions for patients in this observation or clinical decision unit? Capp et al., however, did not include this variable to further stratify the data. Although we do not know whether or not inclusion of this factor may have ultimately changed the results, it could have implications for how distinctions in who manages EDOUs could affect admission rates.

Still, the negative findings of this study seem to raise a number of questions, which should spark a broader discussion on EDOUs. The current analysis provides an important first step toward a national understanding of EDOUs and their role in acute care. Future inquiries should account for variation in observation units and the hospitals in which they are housed as well as inclusion of meaningful outcomes beyond admission rates. A number of methodological approaches can be considered to achieve this; propensity score matching within observational data may provide better balance between facilities with and without EDOUs, whereas multicenter impact analyses using controlled before‐and‐after or cluster‐randomized trials should be considered the gold standard for studying observation unit implementation. Outcomes in these studies should include long‐term changes in health, aggregate healthcare utilization, overuse of resources that do not provide high‐value care, and impacts on how care and costs may be redistributed when patients receive more care in observation units.

Although cost containment is often touted as a cornerstone of EDOUs, it is critical to know how the costs are measured and who is paying. For example, when an option to place a patient in observation exists, might clinicians utilize it for some patients who do not require further evaluation and testing and could have been safely discharged?[9] This observation creep may arise because clinicians can use EDOUs, not because they should. Motivations may include delaying difficult disposition decisions, avoiding uncertainty or liability when discharging patients, limited access to outpatient follow‐up, or a desire to utilize observation status to justify the existence of EDOUs within the institution. In this way, EDOUs may, in fact, provide low‐value care at a time of soaring healthcare costs.

Perhaps even more perplexing is the question of how costs are shifted through use of EDOUs.[10, 11] Much of the literature advertising its cost savings are only from the perspective of the insurers' or hospitals' perspective,[12] with 1 study estimating a potential annual cost savings of $4.6 million for each hospital, or $3 billion nationally, associated with the implementation of observation care.[5] But are medical centers just passing costs on to patients to avoid penalties and disincentives associated with short‐stay hospitalizations? Both private insurers and the Centers for Medicare and Medicaid Services may deny payments for admissions deemed unnecessary. Further, under the Affordable Care Act, avoiding hospitalizations may mean fewer penalties when Medicare patients later require admission for certain conditions. As such, hospitals may find huge incentives and cost savings associated with observation units. However, using EDOUs to avoid the Medicare readmission penalty may backfire when less‐sick patients requiring care beyond the ED are treated and discharged from observation, leaving more medically complex and ill patients for hospitalization, a group potentially more likely to be rehospitalized within 30 days, making readmission rates appear higher.

Nonetheless, because services provided during observation status are billed as an outpatient visit, patients may be liable for a proportion of the overall visit. In contrast to inpatient stays where, in general, patients owe a single copay for most or all of services rendered, outpatient visits typically involve a la carte billing. When accounting for costs related to professional and facilities fees, medications, laboratory tests, and advanced diagnostics and procedures, patient bills may be markedly higher when they are placed in observation status. This is especially true for patients covered by Medicare, where observation stays are not covered under Part A.

Research will need to simultaneously identify best practices for how EDOUs are implemented and administered while appraising their impact on patient‐centered outcomes and true costs, from multiple perspectives, including the patient, hospital, and healthcare system. There is reason to be optimistic about EDOUs as potentially high‐value components of the acute care delivery system. However, the widespread implementation of observation units with the assumption that it is cost saving to hospitals and insurers, without high‐quality population studies to inform their impact more broadly, may undermine acceptance by patients and health‐policy experts.

Over the past 3 decades, emergency department observation units (EDOUs) have been increasingly implemented in the United States to supplement emergency department (ED) care in a time of increasing patient volume and hospital crowding. Given the limited availability of hospital resources, EDOUs provide emergency clinicians an extended period of time to evaluate and risk‐stratify patients without necessitating difficult‐to‐obtain outpatient follow‐up or a short‐stay hospitalization. Changes in Medicare and insurer reimbursement policies have incentivized the adoption of EDOUs, and now, over one‐third of EDs nationally offer an observation unit.[1]

Much of the observation‐science literature has been condition and institution specific, showing benefits with respect to cost, quality of care, safety, and patient satisfaction.[2, 3, 4, 5] Until now, there had not been a national study on the impact of EDOUs to investigate important outcome: hospital admission rates. Capp and colleagues, using the National Hospital Ambulatory Care Survey (NHAMCS), attempt to answer a very important question: Do EDs with observation units have lower hospital admission rates?[6] To do so, they first standardize admission rates to sociodemographic and clinical features of the patients, while adjusting for hospital‐level factors. Then they compare the risk‐standardized hospital admission rate between EDs with and without an observation unit as reported in the NHAMCS. The authors make creative and elegant use of this publicly available, national dataset to suggest that EDOUs do not decrease hospital admissions.

The authors appropriately identify some limitations of using such data to answer questions where nuanced, countervailing forces drive the outcome of interest. It is important to note the basic statistical premise that the inability to disprove the null hypothesis is not the same thing as proving that the null hypothesis is true. In other words, although this study was not able to detect a difference between admission rates for hospitals with EDOUs and those without, it cannot be absolutely taken to mean that there is no relationship. The authors clearly state that this study was underpowered given that the difference of ED risk‐standardized hospital admission rates was small and therefore is at risk of type II error. In addition, unmeasured confounding may hide a true association between EDOUs and admission rates. Both static and dynamic measures of ED volume, crowding, and boarding, as well as changes in case mix or acuity may drive adoption of EDOUs,[7] while simultaneously associated with risk of hospitalization. Without balance between the EDs with and without observation units, or longitudinal measures of EDs over time as they are implemented, we are left with potentially biased estimates.

It is also important to highlight that not all EDOUs are created equal.[8] EDs may admit patients to the observation unit based on prespecified conditions or include all comers at physician discretion. Once placed in observation status, patients may or may not be managed by specific protocols to provide guidance on timing, order, and scope of testing and decision making.

Finally, care in EDOUs may be provided by emergency physicians, hospitalists, or other clinicians such as advanced practice providers (eg, physician assistants, nurse practitioners), a distinction that likely impacts the ultimate patient disposition. In fact, the NHAMCS asks the question, What type of physicians make decisions for patients in this observation or clinical decision unit? Capp et al., however, did not include this variable to further stratify the data. Although we do not know whether or not inclusion of this factor may have ultimately changed the results, it could have implications for how distinctions in who manages EDOUs could affect admission rates.

Still, the negative findings of this study seem to raise a number of questions, which should spark a broader discussion on EDOUs. The current analysis provides an important first step toward a national understanding of EDOUs and their role in acute care. Future inquiries should account for variation in observation units and the hospitals in which they are housed as well as inclusion of meaningful outcomes beyond admission rates. A number of methodological approaches can be considered to achieve this; propensity score matching within observational data may provide better balance between facilities with and without EDOUs, whereas multicenter impact analyses using controlled before‐and‐after or cluster‐randomized trials should be considered the gold standard for studying observation unit implementation. Outcomes in these studies should include long‐term changes in health, aggregate healthcare utilization, overuse of resources that do not provide high‐value care, and impacts on how care and costs may be redistributed when patients receive more care in observation units.

Although cost containment is often touted as a cornerstone of EDOUs, it is critical to know how the costs are measured and who is paying. For example, when an option to place a patient in observation exists, might clinicians utilize it for some patients who do not require further evaluation and testing and could have been safely discharged?[9] This observation creep may arise because clinicians can use EDOUs, not because they should. Motivations may include delaying difficult disposition decisions, avoiding uncertainty or liability when discharging patients, limited access to outpatient follow‐up, or a desire to utilize observation status to justify the existence of EDOUs within the institution. In this way, EDOUs may, in fact, provide low‐value care at a time of soaring healthcare costs.

Perhaps even more perplexing is the question of how costs are shifted through use of EDOUs.[10, 11] Much of the literature advertising its cost savings are only from the perspective of the insurers' or hospitals' perspective,[12] with 1 study estimating a potential annual cost savings of $4.6 million for each hospital, or $3 billion nationally, associated with the implementation of observation care.[5] But are medical centers just passing costs on to patients to avoid penalties and disincentives associated with short‐stay hospitalizations? Both private insurers and the Centers for Medicare and Medicaid Services may deny payments for admissions deemed unnecessary. Further, under the Affordable Care Act, avoiding hospitalizations may mean fewer penalties when Medicare patients later require admission for certain conditions. As such, hospitals may find huge incentives and cost savings associated with observation units. However, using EDOUs to avoid the Medicare readmission penalty may backfire when less‐sick patients requiring care beyond the ED are treated and discharged from observation, leaving more medically complex and ill patients for hospitalization, a group potentially more likely to be rehospitalized within 30 days, making readmission rates appear higher.

Nonetheless, because services provided during observation status are billed as an outpatient visit, patients may be liable for a proportion of the overall visit. In contrast to inpatient stays where, in general, patients owe a single copay for most or all of services rendered, outpatient visits typically involve a la carte billing. When accounting for costs related to professional and facilities fees, medications, laboratory tests, and advanced diagnostics and procedures, patient bills may be markedly higher when they are placed in observation status. This is especially true for patients covered by Medicare, where observation stays are not covered under Part A.

Research will need to simultaneously identify best practices for how EDOUs are implemented and administered while appraising their impact on patient‐centered outcomes and true costs, from multiple perspectives, including the patient, hospital, and healthcare system. There is reason to be optimistic about EDOUs as potentially high‐value components of the acute care delivery system. However, the widespread implementation of observation units with the assumption that it is cost saving to hospitals and insurers, without high‐quality population studies to inform their impact more broadly, may undermine acceptance by patients and health‐policy experts.

Today more than one‐third of emergency departments (EDs) in the United States have affiliated observation units, where patients can stay 24 to 48 hours without being admitted to the hospital.[1] Observation units experienced significant growth in the United States from 2005 to 2007, secondary to policy changes involving the Centers for Medicare and Medicaid Services (CMS), which expanded reimbursement for observation services to include any clinical condition. Furthermore, CMS implemented the Recovery Audit Contractor process, which was able to fine providers and facilities for inappropriate claims, with the principle method for charge recovery being inappropriate charges for short inpatient stays.

ED observation units (EDOUs) vary in the number of beds, but are often located adjacent to the emergency department.[2] It is estimated that EDOUs have the capacity for caring for 5% to 10% of any given ED volume.[2] Almost half of EDOUs are protocol driven, allowing these units to discharge up to 80% of all patients within 24 hours.[1, 2] Some studies have suggested that EDOUs are associated with a decrease in overall hospitalization rates, leading to cost savings.[1] However, these studies were limited by their single‐center design or simulated in nature. In addition, other studies show that EDOUs decrease inpatient admissions, length of stay, and costs related to specific clinical conditions such as chest pain, transient ischemic attack, and syncope.[3]

To further evaluate the association of observation units on ED hospital admission rates nationally, we analyzed the largest ED‐based survey, the 2010 National Hospital Ambulatory Medical Care Survey (NHAMCS), to assess the impact of observation units on hospital admissions from the ED. We hypothesized that observation units decrease overall hospital admissions from the ED.

METHODS

RESULTS

There were 24,232 ED visits from 315 hospitals in the United States in our study. Of these, 82 (20.6%) hospitals had an observation unit physically separate from the ED. Hospitals with and without observation units did not have different hospital patient level characteristics. There was no association between hospital ownership, teaching status, region location, urban or rural location, and hospitals with observation units when compared with hospitals without observation units (Table 1).

In addition, there was no association between patient characteristics at the ED visit level in hospitals with observation units when compared with patient characteristics at the ED visit level in hospitals without observation units (Table 2). The average ED risk‐standardized hospital admission rate for hospitals with observation units was 13.7% (95% confidence interval [CI]: 11.3 to 16.0) compared to 16.0% (95% CI: 14.1 to 17.7) for hospitals without observation units (Figure 1). This difference of 2.3% (95% CI: 0.1 to 4.7) was not statistically significant.

Figure 1