Ponce BA

Over the past several decades, an increasing number of orthopedic surgery residents have pursued fellowship training.¹ This inclination parallels market trends toward subspecialization.^2-5 In 1984, 83% of orthopedics job announcements were for general orthopedists. Twenty-five years later, almost 70% of orthopedic opportunities were for fellowship-trained surgeons.⁶ Further, between 1990 and 2006, the proportion of practicing orthopedic generalists decreased from 44% to 29%.³ In 2007, the American Academy of Orthopaedic Surgery (AAOS) reported 90% of graduating residents were planning to pursue fellowship training.⁷ Reasons for the explosion in subspecialty training are plentiful and well documented.^2-5 Subspecialty positions now dominate the job market, further reinforcing incentives for residents to pursue fellowship training.

The past several decades have seen numerous changes in the orthopedic fellowship interview process. Early on, it was largely unregulated, dependent on personal and professional connections, and flush with the classic “exploding offer” (residents were given a fellowship offer that expired within hours or days). In the 1980s, as the number of fellowship applications surged, the Accreditation Council for Graduate Medical Education (ACGME) pushed for a more regulated process.⁸ To further standardize the system, the American Orthopaedic Association (AOA), the AAOS, and several other specialty organizations created the Orthopaedic Fellowship Match Program Initiative in 2008.⁹ Currently, all orthopedic specialties are represented in either the San Francisco Match Program or National Residency Match Program.

As the system currently stands, postgraduate year 4 (PGY-4) residents are required to interview across the country to secure postgraduate training. This process necessitates residents’ absence from their program, reducing educational opportunities and placing potential continuity-of-care constraints on the residency program. Despite the growing competitiveness for fellowship positions, the increasing number of fellowships available, the rising educational debt of residents, and the limitations of the 80-hour work week, the impact of the interview process on both residents and residency programs has received minimal attention.

We conducted a study to elucidate the impact of the fellowship interview process on residents and residency programs. We hypothesized the time and financial costs for fellowship interviews would be substantial.

Materials and Methods

We obtained institutional review board (IRB) approval for this study. Then, in April 2014, we sent 2 mixed-response questionnaires to orthopedic surgery residency directors and residents. There were 8 items on the director questionnaire and 11 on the resident questionnaire. The surveys were designed to determine the impact of the fellowship interview process on residents and residency programs with respect to finances, time, education, and continuity of care. Each survey had at least 1 free-response question, providing the opportunity to recommend changes to the interview process. The surveys were reviewed and approved by our IRB.

An email was sent to 155 orthopedic surgery program directors or their secretaries. The email asked that the director complete the director questionnaire and that the resident questionnaire be forwarded to senior-level residents, PGY-4s and PGY-5s, who had completed the fellowship interview process. Forty-five (29%) of the 155 directors responded, as did 129 (estimated 9.5%) of an estimated 1354 potential PGY-4s and PGY-5s.¹⁰

The Survey Monkey surveys could be completed over a 3-week period. All responses were anonymous. Using Survey Monkey, we aggregated individual responses into predefined clusters before performing statistical analysis. Descriptive statistics were generated with Microsoft Excel.

Results

Survey respondents represented all the orthopedic subspecialties (Table). Seventy-eight percent of residents applied to at least 13 programs (average, 19) (Figure 1). Ninety-two percent received at least 8 interview offers (average, 14). Eighty-three percent attended 8 or more interviews (average, 11). Seventy-one percent of all interviews were granted when requested, and 79% of all interviews were attended when offered.

Residents spent an average of $5875 (range, $500-$12,000+) on the fellowship interview process (Figure 2). The highest percentage of respondents, 39.5%, selected an average expense between $4000 and $6000. Forty-nine percent of residents borrowed money (from credit cards, additional loans, family members) to pay their expenses.

Average number of days away from residency programs was 11, with 86% of residents missing more than 8 days (Figure 1). About one-third of residents reported being away from their home program for almost 2 weeks during the interview season. Further, 74% of residents wanted changes made to the fellowship application process.

Thirty-seven (82%) of the 45 program directors were from academic programs, the other 8 from community-based programs. Average number of residents in programs per year was 4 (73% of the programs had 4-6 residents per year). Respondents rated the disruption caused by residents’ interview absences from 1 (least disruptive) to 10 (most disruptive) (Figure 3); the average rating was over 7 (high level of disruption). Although 9% of directors thought the process caused little or no disruption (rating, 1-3), 62% thought it extremely disruptive (rating, 8-10).

Thirty-one (69%) of the 45 directors agreed that the fellowship interview process should undergo fundamental change. Asked about possible solutions to current complaints, 60% of the directors agreed that interviews should be conducted in a central location. Of the directors who thought fundamental change was needed, 59% indicated AAOS and other specialty societies together should lead the change in the fellowship interview process.

Both residents and program directors were given the opportunity to write in suggestions regarding how to improve the fellowship interview process. Suggestions were made by 85 (66%) of the 129 residents and 24 (53%) of the 45 directors (Appendix).        

Discussion

Graduating residents are entering a health care environment in which they must be financially conscious because of increasing education debt and decreasing reimbursement prospects.³ Nevertheless, an overwhelming majority of residents delay entering practice to pursue fellowship training—an estimated opportunity cost of $350,000.³ Minimal attention has been given to the potential costs of the fellowship interview process.

Our study results highlight that time away from residency training, financial costs associated with the fellowship interview process, and disruption of the residency program are substantial. On average, residents applied to 19 programs, received 14 interview offers, attended 11 interviews, were away from residency training 11 days, and spent $5875 on travel. The great majority of both residents and program directors wanted changes in the current paradigm governing the orthopedic fellowship interview process.

It is reasonable to think that the number of days residents spend away on interviews would reduce the time available for education and patient care. Although unknown, it is plausible that residents of programs outside major metropolitan centers and residents who apply to more competitive fellowships may be forced to spend even more time away from training. Outside the focus of this study are the impact that residents’ absence might have on their education and the impact of this absence on the people who do the residents’ work while they are away.

Mean fellowship expense was similar to that reported by residents pursuing a pediatric general surgery fellowship ($6974) or a plastic surgery fellowship ($6100).^11,12 Unfortunately, we were unable to determine if average cost is influenced by choice of fellowship specialty or location of residency program. Regardless, fellowship cost may impose an additional financial burden on residents. According to the Association of American Medical Colleges (AAMC), the median salary for PGY-4 residents was $56,380 in 2013. Therefore, on average, the fellowship process consumes more than 10% of a resident’s pretax salary. For perspective, this equates to more than $40,000 for a practicing orthopedic surgeon with a median salary of $413,000.¹³ With an average medical student graduate debt of $175,000 and continuing decreases in reimbursement, further financial hardships to newly graduating residents cannot be understated.^5,11,12

Almost 70% of program directors thought the fellowship process significantly disrupted their program. Reasons given for this disruption mainly involved residents’ time away from the program and the resulting strains placed on maintaining adequate coverage for patient care. The overall disruption score of 7.4 out of 10 was consistent with the great majority thinking that the fellowship process negatively affects their residency program. Altering the fellowship interview process may provide unintended benefits to programs and program directors.

Both program directors and residents communicated that change is needed, but there was little consensus regarding how to effect change and who should lead. This lack of consensus highlights how important it is for the various orthopedic leadership committees to actively and collectively participate in discussions about redefining the system. It has been proposed that it would be ideal for the AOA to lead the change, as the AOA consists of a representative cohort of academic orthopedists and leaders across the spectrum of all fellowship specialties.¹⁴ Given the abundant concern of both residents and program directors, we find it prudent to issue a call to arms of sorts to the AAOS and the individual orthopedic subspecialty societies to work together on a common goal that would benefit residents, programs, and subspecialties within orthopedics.

In trying to understand the challenges that residents, program directors, and programs face, as well as the inherent complexity of the current system, we incorporated respondents’ write-in comments into suggested ways of improving the fellowship interview process. These comments had broad perspectives but overall were consistent with the survey results (Appendix).

Technology

Health care is continually finding new ways to take advantage of technological advances. This is occurring with the fellowship interview schema. Numerous disciplines are using videoconferencing platforms (eg, Skype) to conduct interviews. This practice is becoming more commonplace in the business sector. In a recent survey, more than 60% of human resource managers reported conducting video interviews.¹⁵ Two independent residency programs have used video interviews with mixed success.^16,17

Another technological change requested by residents is the creation and updating of fellowship web pages with standardized information. Such a service may prove useful to residents researching a program and may even lead to limiting the number of programs residents apply to, as they may be able to dial in on exactly what distinguishes one program from another before traveling for an interview. A recent study of orthopedic sports medicine fellowship programs found that most of these programs lacked pertinent information on their websites.¹⁸ Important information regarding case logs from current and former fellows; number of faculty, residents, and fellows; and schedules and facilities of interview sites are a few of the online data points that may help residents differentiate particular programs.^19,20 Questions like these are often asked at interviews and site visits. Having accurate information easily available online may reduce or eliminate the need to travel to a site for such information. Standardizing information would also increase transparency among available fellowships. Although not specifically mentioned, organizational software that improves the productivity of the process may help limit the large number of programs applied to, the interviews offered and attended, the days away, and the financial costs without reducing the match rate.

Timing and Location

The issue of timing—with respect to geographical or meteorological concerns—was another recurring theme among respondents. Numerous respondents indicated that certain programs located in geographic proximity tried to minimize travel by offering interviews around the same time. This coordination potentially minimizes travel expenses and time away from the residency program by allowing residents to interview at multiple locations during a single trip per region. The sports medicine fellowship process was identified as a good example of aligning interviews based on geography. Several respondents suggested an option that also reflects the practice of nonsurgical fellowships—delaying the interview season to bypass potential weather concerns. Winter 2013–2014 saw the most flight delays or cancellations in more than a decade; about 50% of all flights scheduled between December and February were delayed or canceled.²¹ Beyond the additional factor of more time away or missing an interview because of the weather are safety concerns related to the weather. One resident reported having a motor vehicle accident while traveling to an interview in poor weather conditions (Appendix).

National Meetings

Each orthopedic subspecialty has numerous national meetings. Many programs offer applicants the opportunity to interview at these meetings. One respondent mentioned that the annual meeting of the Orthopaedic Trauma Association offers trauma applicants the opportunity to interview with multiple programs. It might be beneficial to endorse this practice on a larger scale to help reduce travel and time away. We recognize that visiting individual programs is an important aspect of the match process, but doing so on a targeted level may make more sense, increasing financial efficiency and reducing time away from programs.

Proposed Solution

A combined proposed solution that can be implemented without a radical overhaul or significant investments might involve moving the interview season to early spring, switching to a 2-tiered system with a centralized first round of interview screening coinciding with subspecialty national meetings or the AAOS annual meeting, and standardizing online information for all orthopedic fellowship programs. A 2-tiered interview process would allow programs and candidates to obtain exposure to a significant number of programs in the first round without incurring significant costs and then would impose a cap on the number of programs to visit. This would level the playing field between candidates with more time and money and candidates who are more constrained in their training environment and finances. A stopgap or adjunct to residents or fellowship programs unable to attend a centralized meeting would be to combine technological tools, such as Internet-based videoconferencing (Skype), before site visits by residents. After this first round of introductions and interviews, residents could then decide on a limited number of programs to formally visit, attend, and ultimately rank. This proposed system would still be able to function within the confines of the match, and it would benefit from the protections offered to residents and programs. Although capping the number of interviews attended by residents clearly can lower costs across the board, we recognize the difficulty of enforcing such a requirement. These potential changes to the system are not exhaustive, and we hope this work will serve as a springboard to further discussion.

Our study had several inherent weaknesses. Our data came from survey responses, which reflect the perspectives only of the responding residents and program directors. Unfortunately, a small number of orthopedic residents responded to this survey, so there was a potential for bias. However, we think the central themes discovered in this survey are only echoes of the concerns of the larger population of residents and program directors. Our hope in designing such a study was to bring to light some of the discrepancies in the fellowship interview process, the goal being to stimulate interest among the orthopedic leadership representing future orthopedic surgeons. More study is needed to clarify if these issues are reflective of a larger segment of residents and program directors. In addition, action may be needed to fully elucidate the intricate interworking of the fellowship process in order to maximize the interest of the orthopedic surgeons who are seeking fellowship training. Another study limitation was the potential for recall bias in the more senior PGY-5 residents, who were further from the interview process than PGY-4 respondents were. Because of the need for anonymity with the surveys, we could not link some findings (eg, program impact, cost, time away) to individual programs or different specialty fellowships. Although it appears there is a desire for a more cost-effective system, given the financial pressures on medical students and residents, the desire to match increases costs because students are likely to attend more interviews than actually needed. Our proposed solution does not take into account residents’ behavior with respect to the current match system. For example, the prevailing thought is that interviewing at more programs increases the likelihood of matching into a desired subspecialty. Despite these study limitations, we think our results identified important points for discussion, investigation, and potential action by orthopedic leadership.

Conclusion

The challenge of critiquing and improving the orthopedic fellowship process requires the same courageous leadership that was recommended almost a decade ago.¹⁴ In this study, we tried to elucidate the impact of the PGY-4 fellowship interview process with respect to residents and residency programs. Our results highlight that time away from residency training, financial costs associated with the fellowship interview process, and disruption of the residency program are substantial and that both residents and program directors want changes made. Leadership needs to further investigate alternatives to the current process to lessen the impact on all parties in this important process.

Over the past several decades, an increasing number of orthopedic surgery residents have pursued fellowship training.¹ This inclination parallels market trends toward subspecialization.^2-5 In 1984, 83% of orthopedics job announcements were for general orthopedists. Twenty-five years later, almost 70% of orthopedic opportunities were for fellowship-trained surgeons.⁶ Further, between 1990 and 2006, the proportion of practicing orthopedic generalists decreased from 44% to 29%.³ In 2007, the American Academy of Orthopaedic Surgery (AAOS) reported 90% of graduating residents were planning to pursue fellowship training.⁷ Reasons for the explosion in subspecialty training are plentiful and well documented.^2-5 Subspecialty positions now dominate the job market, further reinforcing incentives for residents to pursue fellowship training.

The past several decades have seen numerous changes in the orthopedic fellowship interview process. Early on, it was largely unregulated, dependent on personal and professional connections, and flush with the classic “exploding offer” (residents were given a fellowship offer that expired within hours or days). In the 1980s, as the number of fellowship applications surged, the Accreditation Council for Graduate Medical Education (ACGME) pushed for a more regulated process.⁸ To further standardize the system, the American Orthopaedic Association (AOA), the AAOS, and several other specialty organizations created the Orthopaedic Fellowship Match Program Initiative in 2008.⁹ Currently, all orthopedic specialties are represented in either the San Francisco Match Program or National Residency Match Program.

As the system currently stands, postgraduate year 4 (PGY-4) residents are required to interview across the country to secure postgraduate training. This process necessitates residents’ absence from their program, reducing educational opportunities and placing potential continuity-of-care constraints on the residency program. Despite the growing competitiveness for fellowship positions, the increasing number of fellowships available, the rising educational debt of residents, and the limitations of the 80-hour work week, the impact of the interview process on both residents and residency programs has received minimal attention.

We conducted a study to elucidate the impact of the fellowship interview process on residents and residency programs. We hypothesized the time and financial costs for fellowship interviews would be substantial.

Materials and Methods

We obtained institutional review board (IRB) approval for this study. Then, in April 2014, we sent 2 mixed-response questionnaires to orthopedic surgery residency directors and residents. There were 8 items on the director questionnaire and 11 on the resident questionnaire. The surveys were designed to determine the impact of the fellowship interview process on residents and residency programs with respect to finances, time, education, and continuity of care. Each survey had at least 1 free-response question, providing the opportunity to recommend changes to the interview process. The surveys were reviewed and approved by our IRB.

An email was sent to 155 orthopedic surgery program directors or their secretaries. The email asked that the director complete the director questionnaire and that the resident questionnaire be forwarded to senior-level residents, PGY-4s and PGY-5s, who had completed the fellowship interview process. Forty-five (29%) of the 155 directors responded, as did 129 (estimated 9.5%) of an estimated 1354 potential PGY-4s and PGY-5s.¹⁰

The Survey Monkey surveys could be completed over a 3-week period. All responses were anonymous. Using Survey Monkey, we aggregated individual responses into predefined clusters before performing statistical analysis. Descriptive statistics were generated with Microsoft Excel.

Results

Survey respondents represented all the orthopedic subspecialties (Table). Seventy-eight percent of residents applied to at least 13 programs (average, 19) (Figure 1). Ninety-two percent received at least 8 interview offers (average, 14). Eighty-three percent attended 8 or more interviews (average, 11). Seventy-one percent of all interviews were granted when requested, and 79% of all interviews were attended when offered.

Residents spent an average of $5875 (range, $500-$12,000+) on the fellowship interview process (Figure 2). The highest percentage of respondents, 39.5%, selected an average expense between $4000 and $6000. Forty-nine percent of residents borrowed money (from credit cards, additional loans, family members) to pay their expenses.

Average number of days away from residency programs was 11, with 86% of residents missing more than 8 days (Figure 1). About one-third of residents reported being away from their home program for almost 2 weeks during the interview season. Further, 74% of residents wanted changes made to the fellowship application process.

Thirty-seven (82%) of the 45 program directors were from academic programs, the other 8 from community-based programs. Average number of residents in programs per year was 4 (73% of the programs had 4-6 residents per year). Respondents rated the disruption caused by residents’ interview absences from 1 (least disruptive) to 10 (most disruptive) (Figure 3); the average rating was over 7 (high level of disruption). Although 9% of directors thought the process caused little or no disruption (rating, 1-3), 62% thought it extremely disruptive (rating, 8-10).

Thirty-one (69%) of the 45 directors agreed that the fellowship interview process should undergo fundamental change. Asked about possible solutions to current complaints, 60% of the directors agreed that interviews should be conducted in a central location. Of the directors who thought fundamental change was needed, 59% indicated AAOS and other specialty societies together should lead the change in the fellowship interview process.

Both residents and program directors were given the opportunity to write in suggestions regarding how to improve the fellowship interview process. Suggestions were made by 85 (66%) of the 129 residents and 24 (53%) of the 45 directors (Appendix).        

Discussion

Graduating residents are entering a health care environment in which they must be financially conscious because of increasing education debt and decreasing reimbursement prospects.³ Nevertheless, an overwhelming majority of residents delay entering practice to pursue fellowship training—an estimated opportunity cost of $350,000.³ Minimal attention has been given to the potential costs of the fellowship interview process.

Our study results highlight that time away from residency training, financial costs associated with the fellowship interview process, and disruption of the residency program are substantial. On average, residents applied to 19 programs, received 14 interview offers, attended 11 interviews, were away from residency training 11 days, and spent $5875 on travel. The great majority of both residents and program directors wanted changes in the current paradigm governing the orthopedic fellowship interview process.

It is reasonable to think that the number of days residents spend away on interviews would reduce the time available for education and patient care. Although unknown, it is plausible that residents of programs outside major metropolitan centers and residents who apply to more competitive fellowships may be forced to spend even more time away from training. Outside the focus of this study are the impact that residents’ absence might have on their education and the impact of this absence on the people who do the residents’ work while they are away.

Mean fellowship expense was similar to that reported by residents pursuing a pediatric general surgery fellowship ($6974) or a plastic surgery fellowship ($6100).^11,12 Unfortunately, we were unable to determine if average cost is influenced by choice of fellowship specialty or location of residency program. Regardless, fellowship cost may impose an additional financial burden on residents. According to the Association of American Medical Colleges (AAMC), the median salary for PGY-4 residents was $56,380 in 2013. Therefore, on average, the fellowship process consumes more than 10% of a resident’s pretax salary. For perspective, this equates to more than $40,000 for a practicing orthopedic surgeon with a median salary of $413,000.¹³ With an average medical student graduate debt of $175,000 and continuing decreases in reimbursement, further financial hardships to newly graduating residents cannot be understated.^5,11,12

Almost 70% of program directors thought the fellowship process significantly disrupted their program. Reasons given for this disruption mainly involved residents’ time away from the program and the resulting strains placed on maintaining adequate coverage for patient care. The overall disruption score of 7.4 out of 10 was consistent with the great majority thinking that the fellowship process negatively affects their residency program. Altering the fellowship interview process may provide unintended benefits to programs and program directors.

Both program directors and residents communicated that change is needed, but there was little consensus regarding how to effect change and who should lead. This lack of consensus highlights how important it is for the various orthopedic leadership committees to actively and collectively participate in discussions about redefining the system. It has been proposed that it would be ideal for the AOA to lead the change, as the AOA consists of a representative cohort of academic orthopedists and leaders across the spectrum of all fellowship specialties.¹⁴ Given the abundant concern of both residents and program directors, we find it prudent to issue a call to arms of sorts to the AAOS and the individual orthopedic subspecialty societies to work together on a common goal that would benefit residents, programs, and subspecialties within orthopedics.

In trying to understand the challenges that residents, program directors, and programs face, as well as the inherent complexity of the current system, we incorporated respondents’ write-in comments into suggested ways of improving the fellowship interview process. These comments had broad perspectives but overall were consistent with the survey results (Appendix).

Technology

Health care is continually finding new ways to take advantage of technological advances. This is occurring with the fellowship interview schema. Numerous disciplines are using videoconferencing platforms (eg, Skype) to conduct interviews. This practice is becoming more commonplace in the business sector. In a recent survey, more than 60% of human resource managers reported conducting video interviews.¹⁵ Two independent residency programs have used video interviews with mixed success.^16,17

Another technological change requested by residents is the creation and updating of fellowship web pages with standardized information. Such a service may prove useful to residents researching a program and may even lead to limiting the number of programs residents apply to, as they may be able to dial in on exactly what distinguishes one program from another before traveling for an interview. A recent study of orthopedic sports medicine fellowship programs found that most of these programs lacked pertinent information on their websites.¹⁸ Important information regarding case logs from current and former fellows; number of faculty, residents, and fellows; and schedules and facilities of interview sites are a few of the online data points that may help residents differentiate particular programs.^19,20 Questions like these are often asked at interviews and site visits. Having accurate information easily available online may reduce or eliminate the need to travel to a site for such information. Standardizing information would also increase transparency among available fellowships. Although not specifically mentioned, organizational software that improves the productivity of the process may help limit the large number of programs applied to, the interviews offered and attended, the days away, and the financial costs without reducing the match rate.

Timing and Location

The issue of timing—with respect to geographical or meteorological concerns—was another recurring theme among respondents. Numerous respondents indicated that certain programs located in geographic proximity tried to minimize travel by offering interviews around the same time. This coordination potentially minimizes travel expenses and time away from the residency program by allowing residents to interview at multiple locations during a single trip per region. The sports medicine fellowship process was identified as a good example of aligning interviews based on geography. Several respondents suggested an option that also reflects the practice of nonsurgical fellowships—delaying the interview season to bypass potential weather concerns. Winter 2013–2014 saw the most flight delays or cancellations in more than a decade; about 50% of all flights scheduled between December and February were delayed or canceled.²¹ Beyond the additional factor of more time away or missing an interview because of the weather are safety concerns related to the weather. One resident reported having a motor vehicle accident while traveling to an interview in poor weather conditions (Appendix).

National Meetings

Each orthopedic subspecialty has numerous national meetings. Many programs offer applicants the opportunity to interview at these meetings. One respondent mentioned that the annual meeting of the Orthopaedic Trauma Association offers trauma applicants the opportunity to interview with multiple programs. It might be beneficial to endorse this practice on a larger scale to help reduce travel and time away. We recognize that visiting individual programs is an important aspect of the match process, but doing so on a targeted level may make more sense, increasing financial efficiency and reducing time away from programs.

Proposed Solution

A combined proposed solution that can be implemented without a radical overhaul or significant investments might involve moving the interview season to early spring, switching to a 2-tiered system with a centralized first round of interview screening coinciding with subspecialty national meetings or the AAOS annual meeting, and standardizing online information for all orthopedic fellowship programs. A 2-tiered interview process would allow programs and candidates to obtain exposure to a significant number of programs in the first round without incurring significant costs and then would impose a cap on the number of programs to visit. This would level the playing field between candidates with more time and money and candidates who are more constrained in their training environment and finances. A stopgap or adjunct to residents or fellowship programs unable to attend a centralized meeting would be to combine technological tools, such as Internet-based videoconferencing (Skype), before site visits by residents. After this first round of introductions and interviews, residents could then decide on a limited number of programs to formally visit, attend, and ultimately rank. This proposed system would still be able to function within the confines of the match, and it would benefit from the protections offered to residents and programs. Although capping the number of interviews attended by residents clearly can lower costs across the board, we recognize the difficulty of enforcing such a requirement. These potential changes to the system are not exhaustive, and we hope this work will serve as a springboard to further discussion.

Our study had several inherent weaknesses. Our data came from survey responses, which reflect the perspectives only of the responding residents and program directors. Unfortunately, a small number of orthopedic residents responded to this survey, so there was a potential for bias. However, we think the central themes discovered in this survey are only echoes of the concerns of the larger population of residents and program directors. Our hope in designing such a study was to bring to light some of the discrepancies in the fellowship interview process, the goal being to stimulate interest among the orthopedic leadership representing future orthopedic surgeons. More study is needed to clarify if these issues are reflective of a larger segment of residents and program directors. In addition, action may be needed to fully elucidate the intricate interworking of the fellowship process in order to maximize the interest of the orthopedic surgeons who are seeking fellowship training. Another study limitation was the potential for recall bias in the more senior PGY-5 residents, who were further from the interview process than PGY-4 respondents were. Because of the need for anonymity with the surveys, we could not link some findings (eg, program impact, cost, time away) to individual programs or different specialty fellowships. Although it appears there is a desire for a more cost-effective system, given the financial pressures on medical students and residents, the desire to match increases costs because students are likely to attend more interviews than actually needed. Our proposed solution does not take into account residents’ behavior with respect to the current match system. For example, the prevailing thought is that interviewing at more programs increases the likelihood of matching into a desired subspecialty. Despite these study limitations, we think our results identified important points for discussion, investigation, and potential action by orthopedic leadership.

Conclusion

The challenge of critiquing and improving the orthopedic fellowship process requires the same courageous leadership that was recommended almost a decade ago.¹⁴ In this study, we tried to elucidate the impact of the PGY-4 fellowship interview process with respect to residents and residency programs. Our results highlight that time away from residency training, financial costs associated with the fellowship interview process, and disruption of the residency program are substantial and that both residents and program directors want changes made. Leadership needs to further investigate alternatives to the current process to lessen the impact on all parties in this important process.

Over the past several decades, an increasing number of orthopedic surgery residents have pursued fellowship training.¹ This inclination parallels market trends toward subspecialization.^2-5 In 1984, 83% of orthopedics job announcements were for general orthopedists. Twenty-five years later, almost 70% of orthopedic opportunities were for fellowship-trained surgeons.⁶ Further, between 1990 and 2006, the proportion of practicing orthopedic generalists decreased from 44% to 29%.³ In 2007, the American Academy of Orthopaedic Surgery (AAOS) reported 90% of graduating residents were planning to pursue fellowship training.⁷ Reasons for the explosion in subspecialty training are plentiful and well documented.^2-5 Subspecialty positions now dominate the job market, further reinforcing incentives for residents to pursue fellowship training.

The past several decades have seen numerous changes in the orthopedic fellowship interview process. Early on, it was largely unregulated, dependent on personal and professional connections, and flush with the classic “exploding offer” (residents were given a fellowship offer that expired within hours or days). In the 1980s, as the number of fellowship applications surged, the Accreditation Council for Graduate Medical Education (ACGME) pushed for a more regulated process.⁸ To further standardize the system, the American Orthopaedic Association (AOA), the AAOS, and several other specialty organizations created the Orthopaedic Fellowship Match Program Initiative in 2008.⁹ Currently, all orthopedic specialties are represented in either the San Francisco Match Program or National Residency Match Program.

As the system currently stands, postgraduate year 4 (PGY-4) residents are required to interview across the country to secure postgraduate training. This process necessitates residents’ absence from their program, reducing educational opportunities and placing potential continuity-of-care constraints on the residency program. Despite the growing competitiveness for fellowship positions, the increasing number of fellowships available, the rising educational debt of residents, and the limitations of the 80-hour work week, the impact of the interview process on both residents and residency programs has received minimal attention.

We conducted a study to elucidate the impact of the fellowship interview process on residents and residency programs. We hypothesized the time and financial costs for fellowship interviews would be substantial.

Materials and Methods

We obtained institutional review board (IRB) approval for this study. Then, in April 2014, we sent 2 mixed-response questionnaires to orthopedic surgery residency directors and residents. There were 8 items on the director questionnaire and 11 on the resident questionnaire. The surveys were designed to determine the impact of the fellowship interview process on residents and residency programs with respect to finances, time, education, and continuity of care. Each survey had at least 1 free-response question, providing the opportunity to recommend changes to the interview process. The surveys were reviewed and approved by our IRB.

An email was sent to 155 orthopedic surgery program directors or their secretaries. The email asked that the director complete the director questionnaire and that the resident questionnaire be forwarded to senior-level residents, PGY-4s and PGY-5s, who had completed the fellowship interview process. Forty-five (29%) of the 155 directors responded, as did 129 (estimated 9.5%) of an estimated 1354 potential PGY-4s and PGY-5s.¹⁰

The Survey Monkey surveys could be completed over a 3-week period. All responses were anonymous. Using Survey Monkey, we aggregated individual responses into predefined clusters before performing statistical analysis. Descriptive statistics were generated with Microsoft Excel.

Results

Survey respondents represented all the orthopedic subspecialties (Table). Seventy-eight percent of residents applied to at least 13 programs (average, 19) (Figure 1). Ninety-two percent received at least 8 interview offers (average, 14). Eighty-three percent attended 8 or more interviews (average, 11). Seventy-one percent of all interviews were granted when requested, and 79% of all interviews were attended when offered.

Residents spent an average of $5875 (range, $500-$12,000+) on the fellowship interview process (Figure 2). The highest percentage of respondents, 39.5%, selected an average expense between $4000 and $6000. Forty-nine percent of residents borrowed money (from credit cards, additional loans, family members) to pay their expenses.

Average number of days away from residency programs was 11, with 86% of residents missing more than 8 days (Figure 1). About one-third of residents reported being away from their home program for almost 2 weeks during the interview season. Further, 74% of residents wanted changes made to the fellowship application process.

Thirty-seven (82%) of the 45 program directors were from academic programs, the other 8 from community-based programs. Average number of residents in programs per year was 4 (73% of the programs had 4-6 residents per year). Respondents rated the disruption caused by residents’ interview absences from 1 (least disruptive) to 10 (most disruptive) (Figure 3); the average rating was over 7 (high level of disruption). Although 9% of directors thought the process caused little or no disruption (rating, 1-3), 62% thought it extremely disruptive (rating, 8-10).

Thirty-one (69%) of the 45 directors agreed that the fellowship interview process should undergo fundamental change. Asked about possible solutions to current complaints, 60% of the directors agreed that interviews should be conducted in a central location. Of the directors who thought fundamental change was needed, 59% indicated AAOS and other specialty societies together should lead the change in the fellowship interview process.

Both residents and program directors were given the opportunity to write in suggestions regarding how to improve the fellowship interview process. Suggestions were made by 85 (66%) of the 129 residents and 24 (53%) of the 45 directors (Appendix).        

Discussion

Graduating residents are entering a health care environment in which they must be financially conscious because of increasing education debt and decreasing reimbursement prospects.³ Nevertheless, an overwhelming majority of residents delay entering practice to pursue fellowship training—an estimated opportunity cost of $350,000.³ Minimal attention has been given to the potential costs of the fellowship interview process.

Our study results highlight that time away from residency training, financial costs associated with the fellowship interview process, and disruption of the residency program are substantial. On average, residents applied to 19 programs, received 14 interview offers, attended 11 interviews, were away from residency training 11 days, and spent $5875 on travel. The great majority of both residents and program directors wanted changes in the current paradigm governing the orthopedic fellowship interview process.

It is reasonable to think that the number of days residents spend away on interviews would reduce the time available for education and patient care. Although unknown, it is plausible that residents of programs outside major metropolitan centers and residents who apply to more competitive fellowships may be forced to spend even more time away from training. Outside the focus of this study are the impact that residents’ absence might have on their education and the impact of this absence on the people who do the residents’ work while they are away.

Mean fellowship expense was similar to that reported by residents pursuing a pediatric general surgery fellowship ($6974) or a plastic surgery fellowship ($6100).^11,12 Unfortunately, we were unable to determine if average cost is influenced by choice of fellowship specialty or location of residency program. Regardless, fellowship cost may impose an additional financial burden on residents. According to the Association of American Medical Colleges (AAMC), the median salary for PGY-4 residents was $56,380 in 2013. Therefore, on average, the fellowship process consumes more than 10% of a resident’s pretax salary. For perspective, this equates to more than $40,000 for a practicing orthopedic surgeon with a median salary of $413,000.¹³ With an average medical student graduate debt of $175,000 and continuing decreases in reimbursement, further financial hardships to newly graduating residents cannot be understated.^5,11,12

Almost 70% of program directors thought the fellowship process significantly disrupted their program. Reasons given for this disruption mainly involved residents’ time away from the program and the resulting strains placed on maintaining adequate coverage for patient care. The overall disruption score of 7.4 out of 10 was consistent with the great majority thinking that the fellowship process negatively affects their residency program. Altering the fellowship interview process may provide unintended benefits to programs and program directors.

Both program directors and residents communicated that change is needed, but there was little consensus regarding how to effect change and who should lead. This lack of consensus highlights how important it is for the various orthopedic leadership committees to actively and collectively participate in discussions about redefining the system. It has been proposed that it would be ideal for the AOA to lead the change, as the AOA consists of a representative cohort of academic orthopedists and leaders across the spectrum of all fellowship specialties.¹⁴ Given the abundant concern of both residents and program directors, we find it prudent to issue a call to arms of sorts to the AAOS and the individual orthopedic subspecialty societies to work together on a common goal that would benefit residents, programs, and subspecialties within orthopedics.

In trying to understand the challenges that residents, program directors, and programs face, as well as the inherent complexity of the current system, we incorporated respondents’ write-in comments into suggested ways of improving the fellowship interview process. These comments had broad perspectives but overall were consistent with the survey results (Appendix).

Technology

Health care is continually finding new ways to take advantage of technological advances. This is occurring with the fellowship interview schema. Numerous disciplines are using videoconferencing platforms (eg, Skype) to conduct interviews. This practice is becoming more commonplace in the business sector. In a recent survey, more than 60% of human resource managers reported conducting video interviews.¹⁵ Two independent residency programs have used video interviews with mixed success.^16,17

Another technological change requested by residents is the creation and updating of fellowship web pages with standardized information. Such a service may prove useful to residents researching a program and may even lead to limiting the number of programs residents apply to, as they may be able to dial in on exactly what distinguishes one program from another before traveling for an interview. A recent study of orthopedic sports medicine fellowship programs found that most of these programs lacked pertinent information on their websites.¹⁸ Important information regarding case logs from current and former fellows; number of faculty, residents, and fellows; and schedules and facilities of interview sites are a few of the online data points that may help residents differentiate particular programs.^19,20 Questions like these are often asked at interviews and site visits. Having accurate information easily available online may reduce or eliminate the need to travel to a site for such information. Standardizing information would also increase transparency among available fellowships. Although not specifically mentioned, organizational software that improves the productivity of the process may help limit the large number of programs applied to, the interviews offered and attended, the days away, and the financial costs without reducing the match rate.

Timing and Location

The issue of timing—with respect to geographical or meteorological concerns—was another recurring theme among respondents. Numerous respondents indicated that certain programs located in geographic proximity tried to minimize travel by offering interviews around the same time. This coordination potentially minimizes travel expenses and time away from the residency program by allowing residents to interview at multiple locations during a single trip per region. The sports medicine fellowship process was identified as a good example of aligning interviews based on geography. Several respondents suggested an option that also reflects the practice of nonsurgical fellowships—delaying the interview season to bypass potential weather concerns. Winter 2013–2014 saw the most flight delays or cancellations in more than a decade; about 50% of all flights scheduled between December and February were delayed or canceled.²¹ Beyond the additional factor of more time away or missing an interview because of the weather are safety concerns related to the weather. One resident reported having a motor vehicle accident while traveling to an interview in poor weather conditions (Appendix).

National Meetings

Each orthopedic subspecialty has numerous national meetings. Many programs offer applicants the opportunity to interview at these meetings. One respondent mentioned that the annual meeting of the Orthopaedic Trauma Association offers trauma applicants the opportunity to interview with multiple programs. It might be beneficial to endorse this practice on a larger scale to help reduce travel and time away. We recognize that visiting individual programs is an important aspect of the match process, but doing so on a targeted level may make more sense, increasing financial efficiency and reducing time away from programs.

Proposed Solution

A combined proposed solution that can be implemented without a radical overhaul or significant investments might involve moving the interview season to early spring, switching to a 2-tiered system with a centralized first round of interview screening coinciding with subspecialty national meetings or the AAOS annual meeting, and standardizing online information for all orthopedic fellowship programs. A 2-tiered interview process would allow programs and candidates to obtain exposure to a significant number of programs in the first round without incurring significant costs and then would impose a cap on the number of programs to visit. This would level the playing field between candidates with more time and money and candidates who are more constrained in their training environment and finances. A stopgap or adjunct to residents or fellowship programs unable to attend a centralized meeting would be to combine technological tools, such as Internet-based videoconferencing (Skype), before site visits by residents. After this first round of introductions and interviews, residents could then decide on a limited number of programs to formally visit, attend, and ultimately rank. This proposed system would still be able to function within the confines of the match, and it would benefit from the protections offered to residents and programs. Although capping the number of interviews attended by residents clearly can lower costs across the board, we recognize the difficulty of enforcing such a requirement. These potential changes to the system are not exhaustive, and we hope this work will serve as a springboard to further discussion.

Our study had several inherent weaknesses. Our data came from survey responses, which reflect the perspectives only of the responding residents and program directors. Unfortunately, a small number of orthopedic residents responded to this survey, so there was a potential for bias. However, we think the central themes discovered in this survey are only echoes of the concerns of the larger population of residents and program directors. Our hope in designing such a study was to bring to light some of the discrepancies in the fellowship interview process, the goal being to stimulate interest among the orthopedic leadership representing future orthopedic surgeons. More study is needed to clarify if these issues are reflective of a larger segment of residents and program directors. In addition, action may be needed to fully elucidate the intricate interworking of the fellowship process in order to maximize the interest of the orthopedic surgeons who are seeking fellowship training. Another study limitation was the potential for recall bias in the more senior PGY-5 residents, who were further from the interview process than PGY-4 respondents were. Because of the need for anonymity with the surveys, we could not link some findings (eg, program impact, cost, time away) to individual programs or different specialty fellowships. Although it appears there is a desire for a more cost-effective system, given the financial pressures on medical students and residents, the desire to match increases costs because students are likely to attend more interviews than actually needed. Our proposed solution does not take into account residents’ behavior with respect to the current match system. For example, the prevailing thought is that interviewing at more programs increases the likelihood of matching into a desired subspecialty. Despite these study limitations, we think our results identified important points for discussion, investigation, and potential action by orthopedic leadership.

Conclusion

The challenge of critiquing and improving the orthopedic fellowship process requires the same courageous leadership that was recommended almost a decade ago.¹⁴ In this study, we tried to elucidate the impact of the PGY-4 fellowship interview process with respect to residents and residency programs. Our results highlight that time away from residency training, financial costs associated with the fellowship interview process, and disruption of the residency program are substantial and that both residents and program directors want changes made. Leadership needs to further investigate alternatives to the current process to lessen the impact on all parties in this important process.

The incidence of shoulder arthroplasty in the United States is increasing annually,^1-3 and the majority of these operations occur in older patients.^4-6 Elderly patients with cardiovascular, pulmonary, cerebral, renal, and hepatic disease are increasingly susceptible to numerous surgical complications.⁴ Myocardial infarction (MI) is a complication that occurs in 0.7% of noncardiac surgeries. This figure increases to 1.1% in patients with coronary artery disease.^7-11 Perioperative MI increases morbidity and mortality,⁸ and perioperative cardiac morbidity is the leading cause of death after anesthesia and surgery.¹² The financial effects of perioperative cardiac morbidity and mortality must also be considered. A 2009 claims analysis study estimated charges associated with a perioperative MI at $15,000 and the cost of cardiac death at $21,909.¹³

Cardiovascular complications are associated with a significant degree of morbidity and mortality in patients who undergo arthroplasty.^14-16 Although studies have elucidated 30- and 90-day morbidity and mortality rates after shoulder arthroplasty, in hip and knee arthroplasty^17-19 little has been done to determine predictors of perioperative MI in a representative database of patients. Given the increasing incidence of shoulder arthroplasty in the United States, the elective nature of this procedure, and the percentage of the US population with cardiovascular risk factors,²⁰ it is important to establish predictors of perioperative MI to ensure patients and physicians have the necessary resources to make informed decisions.

We conducted a study to examine the risk factors for perioperative MI in a large cohort of patients admitted for shoulder arthroplasty to US hospitals. We wanted to evaluate the association between perioperative MI and shoulder arthroplasty with respect to demographics, primary diagnosis, medical comorbidities, and perioperative complications. Specifically, we tested the null hypothesis that, among patients undergoing shoulder arthroplasty, and accounting for confounding variables, there would be no difference in risk factors for patients who have a perioperative MI.

Materials and Methods

This study was exempt from approval by our institutional review board. All data used in this project were deidentified before use.

Nationwide Inpatient Sample (NIS)

The Nationwide Inpatient Sample (NIS), an annual survey of hospitals, is conducted by the Healthcare Cost and Utilization Project (HCUP) and sponsored by the Agency for Healthcare Research and Quality (AHRQ). This database is the largest publicly available all-payer inpatient discharge database in the United States.²¹ Sampling 8 million hospital stays each year, NIS includes information from a representative batch of 20% of US hospitals. In 2011, 46 states and 1045 hospitals contributed information to the database, representing 97% of the US population.²² This large sample allows researchers to analyze a robust set of medical conditions and uncommon treatments. The survey, conducted each year since 1988, includes demographic, clinical, and resource use data.²³ Discharge weight files are provided by NIS to arrive at valid national estimates.

This database is particularly useful because it provides information on up to 25 medical diagnoses and 15 procedures, which are recorded with International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. Researchers can use this database to analyze patient and hospital characteristics as well as inpatient outcomes.^24,25 Numerous studies have used NIS to address pertinent queries across the medical landscape.^22,26

Patient Selection and Analysis

We used NIS to isolate a population of 422,371 adults (≥18 years old) who underwent total shoulder arthroplasty (TSA) or hemi–shoulder arthroplasty (HSA) between January 1, 2002 and December 31, 2011. We then placed the patients in this population into 1 of 2 cohorts. The first cohort had an acute MI during the perioperative period after TSA, and the second, larger cohort did not have an acute MI after TSA. Acute MI was identified using ICD-9-CM code 410.xx. To identify a population of shoulder arthroplasty patients, we included discharges with an ICD-9-CM procedure code of 81.80 or 81.88 (both TSA) or 81.81 (HSA) in the sample. We then considered the degree to which each of 5 variables—primary diagnosis, age, sex, race, and select medical comorbidities—was predictive of in-hospital MI after TSA.

Statistical Analysis

Given the large sample used in this study, normal distribution of data was assumed. Using bivariate analysis, Pearson χ² test for categorical data, and independent-samples t test for continuous data, we compared the nonacute MI and acute MI groups. Multivariable binary logistic regression analyses allowed us to isolate the extent that primary diagnosis, age, sex, race, and medical comorbidities were predictors of acute MI after shoulder arthroplasty. Statistical significance was set at P < .05. SPSS Version 22.0 (SPSS, Chicago, Illinois) was used for all statistical analyses and data modeling.

Results

Between January 1, 2002 and December 31, 2011, an estimated total of 422,371 patients underwent shoulder arthroplasty (59.3% TSA, 40.7% HSA). Of these patients, 1174 (0.28%) had a perioperative MI, and 421,197 (99.72%) did not (Table 1). Patients with a primary diagnosis of proximal humerus fracture (33.8% vs 16.6%; P < .001) or rotator cuff arthropathy (10.1% vs 9.9%; P < .001) were more likely than patients with other diagnoses to have an in-hospital MI.

Our review of the demographics found that patients who underwent shoulder arthroplasty and had a perioperative MI were likely older (75±8.9 years vs 69±11 years; P < .001), Caucasian (94.2% vs 91.9%; P = .002), male (43.2% vs 39.7%; P = .013), in the highest median household income bracket of $63,000 or more (30.8% vs 25.6%; P < .001), and using Medicare (80.9% vs 66.3%; P < .001). They were more likely to be treated in a medical center of medium size (25.6% vs 23.7%; P = .042) or larger (61.8% vs 61.2%; P = .042). MIs occurred more often in urban environments (91.4% vs 88.5%; P = .002) and in HSA patients (55% vs 40.6%; P < .001), resulting in longer hospital  stays (9.4±7.9 days vs 2.7±2.5 days; P < .001) and higher probability of death (6.5% vs 0.1%; P < .001).

We then analyzed the 2 cohorts for medical comorbidities (Table 2). Patients in the MI cohort presented with a significantly higher incidence of congestive heart failure, previous MI, angina pectoris, chronic lung disease, hypertension, diabetes, renal failure, fluid and electrolyte disorders, pulmonary circulatory disease, coagulopathy, and deficiency anemia (P < .001) but not liver disease and obesity. Bivariate analysis of perioperative outcomes (Table 3) indicated that these patients also had a statistically higher rate of numerous other complications: pulmonary embolism (4.9% vs 0.2%; P < .001), pneumonia (15.1% vs 1.2%; P < .001), deep venous thrombosis (2.6% vs 0.2%; P < .001), cerebrovascular event (1.6% vs 0.1%; P < .001), acute renal failure (15.1% vs 1.2%; P < .001), gastrointestinal complication (1.2% vs 0.3%; P < .001), mechanical ventilation (1.2% vs 0.3%; P < .001), transfusion (33.4% vs 8.8%; P < .001), and nonroutine discharge (73.3% vs 36.0%; P < .001).

Multivariable logistic regression analysis was performed to determine independent predictors of perioperative MI after shoulder arthroplasty (Table 4). Patients with a primary diagnosis of proximal humerus fracture (odds ratio [OR], 1.38; 95% confidence interval [CI], 1.15-1.65; P < .001) were more likely than patients with a primary diagnosis of osteoarthritis to have an MI. The odds of postoperative MI increased with age (OR, 1.04 per year; 95% CI, 1.03-1.05; P < .001) and were higher in males (OR, 1.72; 95% CI, 1.52-1.96; P < .001). Compared with Caucasians, African Americans (OR, 0.19; 95% CI, 0.09-0.40; P < .001) were less likely to have an in-hospital MI after shoulder arthroplasty. After shoulder arthroplasty, the odds of MI in the perioperative period increased with each subsequent day of care (OR, 1.10; 95% CI, 1.10-1.11; P < .001).

Regarding independent comorbidities, multivariable logistic regression analysis also determined that history of congestive heart failure (OR, 4.86; 95% CI, 4.20-5.61; P < .001), angina pectoris (OR, 2.90; 95% CI, 2.02-4.17; P < .001), complicated diabetes (OR, 1.96; 95% CI, 1.49-2.57; P < .001), renal failure (OR, 1.42; 95% CI, 1.17-1.72; P < .001), fluid and electrolyte disorders (OR, 1.42; 95% CI, 1.21-1.67; P < .001), and deficiency anemia (OR, 1.62; 95% CI, 1.40-1.88; P < .001) were significant predictors of perioperative MI after shoulder arthroplasty.

Discussion

Results of other studies have elucidated 30- and 90-day mortality rates and postoperative complications after shoulder arthroplasty, but, relative to hip and knee arthroplasty,^17-19 little has been done to determine predictors of perioperative MI in a large sample of shoulder arthroplasty patients. Given the increasing rates of shoulder arthroplasty^1-3 and the demographics of this population,^4-6 it is likely that postoperative cardiovascular events will increase in frequency. We found that, in order of decreasing significance, the top 4 risk predictors for acute MI after shoulder arthroplasty were congestive heart failure, angina pectoralis, complicated diabetes mellitus, and male sex. Other pertinent risk factors included older age, Caucasian ethnicity, and a primary diagnosis of proximal humerus fracture. The rate of acute MI in patients who were older than 75 years when they underwent HSA for proximal humerus fracture was 0.80%.

Demographics

We found that patients who had an acute MI after shoulder arthroplasty were likely older, male, and Caucasian. Age and male sex are well-established risk factors for increased cardiac complications after arthroplasty.^27-29 Previous studies have indicated that the rate of cardiac events increases in arthroplasty patients older than 65 years.^19,28,29 In our study, more than 50% of the patients who had an acute perioperative MI were older than 85 years. Less explainable is the increased occurrence of acute MI in Caucasian patients and wealthy patients, given that minorities in the United States have higher rates of cardiovascular disease.³⁰ Shoulder arthroplasty is an elective procedure, more likely to be undertaken by Caucasians. Therefore, at-risk minority groups and financially challenged groups may be less likely to have this procedure.

Primary Diagnosis

In this series, patients with a primary diagnosis of proximal humerus fracture were more likely to have an in-hospital MI. This finding is consistent with previous studies indicating a higher rate of complications for proximal humerus fracture patients than for shoulder arthroplasty patients.^31,32 Given that more than 75% of patients who present with a proximal humerus fracture are older than 70 years, it would be prudent to examine operative indications after this diagnosis,³³ particularly as benefit from surgery for fractures has not been definitively demonstrated.^34-37

Comorbidities

Many of the patients in our MI cohort presented with congestive heart failure, angina pectoris, complicated diabetes, renal failure, fluid and electrolyte disorders, or deficiency anemia. This is in keeping with other studies indicating that preexisting cardiovascular morbidity increases the rate of MI after various forms of arthroplasty.^7-11 Patients in our MI cohort were also susceptible to a variety of post-MI perioperative complications, including pulmonary embolism, pneumonia, deep venous thrombosis, cerebrovascular event, acute renal failure, gastrointestinal complication, mechanical ventilation, transfusion, and nonroutine discharge, and their incidence of death was higher. These findings are consistent with reports that postoperative cardiovascular complications increase the degree of morbidity and mortality in arthroplasty patients.^14-16 It is also worth noting that the odds of MI in the perioperative period increase with each subsequent day of care. This is understandable given that patients presenting with numerous comorbidities are at increased risk for perioperative complications³⁸ resulting in hospital readmission.³⁹

The literature indicates that MI occurs as a complication in 0.7% of patients who undergo noncardiac surgery,⁷ though some series have shown it is more prevalent after arthroplasty procedures.^28,40 MI significantly increases the rate of perioperative morbidity and mortality,⁸ and perioperative cardiac morbidity is a leading cause of death after anesthesia and surgery.¹² Furthermore, the most common cause of death after lower extremity arthroplasty is cardiovascular-related.^41,42 In patients who presented for elective hip arthroplasty, cardiorespiratory disease was one of the main risk factors (with older age and male sex) shown to increase perioperative mortality.⁴³

Perioperative cardiovascular complications increase postoperative morbidity and mortality.¹² The rate of cardiovascular complications after shoulder arthroplasty ranges from 0.8% to 2.6%, and the incidence of MI hovers between 0.3% and 0.9%.^{17,19,28,40,44} A recent study in 793 patients found that, over a 30-day period, cardiovascular complications accounted for more than one-fourth of all complications.¹⁷ Singh and colleagues¹⁹ analyzed cardiopulmonary complications after primary shoulder arthroplasty in a total of 3480 patients (4019 arthroplasties) and found this group had a 90-day cardiac morbidity (MI, congestive heart failure, arrhythmia) rate of 2.6%. In that study, a Deyo-Charlson index of 1 or more was a significant independent risk factor for cardiac complications following surgery. Scores on this weighted index of 17 comorbidities are used to assess the complexities of a patient population. Given the severity of cardiovascular perioperative complications, it is important to preoperatively identify high-risk population groups and sufficiently study and optimize patients before shoulder arthroplasty.

There is much debate about the effectiveness of perioperative β-blockers in reducing perioperative cardiac morbidity and mortality.^45-48 Such a discussion is outside of the scope of this article, but it may be prudent to seek a cardiology consultation for patients presenting with risk factors for perioperative MI. β-Blockers may prove useful in reducing cardiac morbidity in high-risk patients after noncardiac surgery.^45,49

Many limitations are inherent in studies that use a nationally represented database such as NIS, which we used in this study. It is highly likely that NIS does not capture all potential postoperative complications, as this database is very large and subject to errors in data entry and clinical coding. In addition, detailed clinical information (eg, severity of certain comorbid diseases before shoulder arthroplasty, details about the intraoperative course) was not readily available for analysis. Another limitation, which may have led to an underestimate of complication rates, was our not being able to obtain information about postdischarge complications.

Despite these limitations, NIS and other databases have helped researchers answer questions about low-incidence conditions and generalize findings to a national population. In the present study, we analyzed 2 cohorts, patients with and without acute MI after shoulder arthroplasty, to determine predictors for and complications of postarthroplasty MI. We identified numerous predictors for acute MI: congestive heart failure, angina pectoris, complicated diabetes, renal failure, fluid and electrolyte disorders, and deficiency anemia prior to arthroplasty. As perioperative MI is associated with significant morbidity,^14-16 it would be wise to screen patients for such comorbid conditions, assess the severity of these conditions, and offer shoulder arthroplasty with prudence.

Conclusion

The top 4 predictors for acute MI after shoulder arthroplasty were congestive heart failure, angina pectoralis, complicated diabetes mellitus, and male sex. Other pertinent risk factors included older age, Caucasian ethnicity, and primary diagnosis of proximal humerus fracture. Surgeons and patients must be aware of predictors for adverse surgical outcomes such as perioperative MI and understand the extent to which these events increase perioperative morbidity and mortality.

The incidence of shoulder arthroplasty in the United States is increasing annually,^1-3 and the majority of these operations occur in older patients.^4-6 Elderly patients with cardiovascular, pulmonary, cerebral, renal, and hepatic disease are increasingly susceptible to numerous surgical complications.⁴ Myocardial infarction (MI) is a complication that occurs in 0.7% of noncardiac surgeries. This figure increases to 1.1% in patients with coronary artery disease.^7-11 Perioperative MI increases morbidity and mortality,⁸ and perioperative cardiac morbidity is the leading cause of death after anesthesia and surgery.¹² The financial effects of perioperative cardiac morbidity and mortality must also be considered. A 2009 claims analysis study estimated charges associated with a perioperative MI at $15,000 and the cost of cardiac death at $21,909.¹³

Cardiovascular complications are associated with a significant degree of morbidity and mortality in patients who undergo arthroplasty.^14-16 Although studies have elucidated 30- and 90-day morbidity and mortality rates after shoulder arthroplasty, in hip and knee arthroplasty^17-19 little has been done to determine predictors of perioperative MI in a representative database of patients. Given the increasing incidence of shoulder arthroplasty in the United States, the elective nature of this procedure, and the percentage of the US population with cardiovascular risk factors,²⁰ it is important to establish predictors of perioperative MI to ensure patients and physicians have the necessary resources to make informed decisions.

We conducted a study to examine the risk factors for perioperative MI in a large cohort of patients admitted for shoulder arthroplasty to US hospitals. We wanted to evaluate the association between perioperative MI and shoulder arthroplasty with respect to demographics, primary diagnosis, medical comorbidities, and perioperative complications. Specifically, we tested the null hypothesis that, among patients undergoing shoulder arthroplasty, and accounting for confounding variables, there would be no difference in risk factors for patients who have a perioperative MI.

Materials and Methods

This study was exempt from approval by our institutional review board. All data used in this project were deidentified before use.

Nationwide Inpatient Sample (NIS)

The Nationwide Inpatient Sample (NIS), an annual survey of hospitals, is conducted by the Healthcare Cost and Utilization Project (HCUP) and sponsored by the Agency for Healthcare Research and Quality (AHRQ). This database is the largest publicly available all-payer inpatient discharge database in the United States.²¹ Sampling 8 million hospital stays each year, NIS includes information from a representative batch of 20% of US hospitals. In 2011, 46 states and 1045 hospitals contributed information to the database, representing 97% of the US population.²² This large sample allows researchers to analyze a robust set of medical conditions and uncommon treatments. The survey, conducted each year since 1988, includes demographic, clinical, and resource use data.²³ Discharge weight files are provided by NIS to arrive at valid national estimates.

This database is particularly useful because it provides information on up to 25 medical diagnoses and 15 procedures, which are recorded with International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. Researchers can use this database to analyze patient and hospital characteristics as well as inpatient outcomes.^24,25 Numerous studies have used NIS to address pertinent queries across the medical landscape.^22,26

Patient Selection and Analysis

We used NIS to isolate a population of 422,371 adults (≥18 years old) who underwent total shoulder arthroplasty (TSA) or hemi–shoulder arthroplasty (HSA) between January 1, 2002 and December 31, 2011. We then placed the patients in this population into 1 of 2 cohorts. The first cohort had an acute MI during the perioperative period after TSA, and the second, larger cohort did not have an acute MI after TSA. Acute MI was identified using ICD-9-CM code 410.xx. To identify a population of shoulder arthroplasty patients, we included discharges with an ICD-9-CM procedure code of 81.80 or 81.88 (both TSA) or 81.81 (HSA) in the sample. We then considered the degree to which each of 5 variables—primary diagnosis, age, sex, race, and select medical comorbidities—was predictive of in-hospital MI after TSA.

Statistical Analysis

Given the large sample used in this study, normal distribution of data was assumed. Using bivariate analysis, Pearson χ² test for categorical data, and independent-samples t test for continuous data, we compared the nonacute MI and acute MI groups. Multivariable binary logistic regression analyses allowed us to isolate the extent that primary diagnosis, age, sex, race, and medical comorbidities were predictors of acute MI after shoulder arthroplasty. Statistical significance was set at P < .05. SPSS Version 22.0 (SPSS, Chicago, Illinois) was used for all statistical analyses and data modeling.

Results

Between January 1, 2002 and December 31, 2011, an estimated total of 422,371 patients underwent shoulder arthroplasty (59.3% TSA, 40.7% HSA). Of these patients, 1174 (0.28%) had a perioperative MI, and 421,197 (99.72%) did not (Table 1). Patients with a primary diagnosis of proximal humerus fracture (33.8% vs 16.6%; P < .001) or rotator cuff arthropathy (10.1% vs 9.9%; P < .001) were more likely than patients with other diagnoses to have an in-hospital MI.

Our review of the demographics found that patients who underwent shoulder arthroplasty and had a perioperative MI were likely older (75±8.9 years vs 69±11 years; P < .001), Caucasian (94.2% vs 91.9%; P = .002), male (43.2% vs 39.7%; P = .013), in the highest median household income bracket of $63,000 or more (30.8% vs 25.6%; P < .001), and using Medicare (80.9% vs 66.3%; P < .001). They were more likely to be treated in a medical center of medium size (25.6% vs 23.7%; P = .042) or larger (61.8% vs 61.2%; P = .042). MIs occurred more often in urban environments (91.4% vs 88.5%; P = .002) and in HSA patients (55% vs 40.6%; P < .001), resulting in longer hospital  stays (9.4±7.9 days vs 2.7±2.5 days; P < .001) and higher probability of death (6.5% vs 0.1%; P < .001).

We then analyzed the 2 cohorts for medical comorbidities (Table 2). Patients in the MI cohort presented with a significantly higher incidence of congestive heart failure, previous MI, angina pectoris, chronic lung disease, hypertension, diabetes, renal failure, fluid and electrolyte disorders, pulmonary circulatory disease, coagulopathy, and deficiency anemia (P < .001) but not liver disease and obesity. Bivariate analysis of perioperative outcomes (Table 3) indicated that these patients also had a statistically higher rate of numerous other complications: pulmonary embolism (4.9% vs 0.2%; P < .001), pneumonia (15.1% vs 1.2%; P < .001), deep venous thrombosis (2.6% vs 0.2%; P < .001), cerebrovascular event (1.6% vs 0.1%; P < .001), acute renal failure (15.1% vs 1.2%; P < .001), gastrointestinal complication (1.2% vs 0.3%; P < .001), mechanical ventilation (1.2% vs 0.3%; P < .001), transfusion (33.4% vs 8.8%; P < .001), and nonroutine discharge (73.3% vs 36.0%; P < .001).

Multivariable logistic regression analysis was performed to determine independent predictors of perioperative MI after shoulder arthroplasty (Table 4). Patients with a primary diagnosis of proximal humerus fracture (odds ratio [OR], 1.38; 95% confidence interval [CI], 1.15-1.65; P < .001) were more likely than patients with a primary diagnosis of osteoarthritis to have an MI. The odds of postoperative MI increased with age (OR, 1.04 per year; 95% CI, 1.03-1.05; P < .001) and were higher in males (OR, 1.72; 95% CI, 1.52-1.96; P < .001). Compared with Caucasians, African Americans (OR, 0.19; 95% CI, 0.09-0.40; P < .001) were less likely to have an in-hospital MI after shoulder arthroplasty. After shoulder arthroplasty, the odds of MI in the perioperative period increased with each subsequent day of care (OR, 1.10; 95% CI, 1.10-1.11; P < .001).

Regarding independent comorbidities, multivariable logistic regression analysis also determined that history of congestive heart failure (OR, 4.86; 95% CI, 4.20-5.61; P < .001), angina pectoris (OR, 2.90; 95% CI, 2.02-4.17; P < .001), complicated diabetes (OR, 1.96; 95% CI, 1.49-2.57; P < .001), renal failure (OR, 1.42; 95% CI, 1.17-1.72; P < .001), fluid and electrolyte disorders (OR, 1.42; 95% CI, 1.21-1.67; P < .001), and deficiency anemia (OR, 1.62; 95% CI, 1.40-1.88; P < .001) were significant predictors of perioperative MI after shoulder arthroplasty.

Discussion

Results of other studies have elucidated 30- and 90-day mortality rates and postoperative complications after shoulder arthroplasty, but, relative to hip and knee arthroplasty,^17-19 little has been done to determine predictors of perioperative MI in a large sample of shoulder arthroplasty patients. Given the increasing rates of shoulder arthroplasty^1-3 and the demographics of this population,^4-6 it is likely that postoperative cardiovascular events will increase in frequency. We found that, in order of decreasing significance, the top 4 risk predictors for acute MI after shoulder arthroplasty were congestive heart failure, angina pectoralis, complicated diabetes mellitus, and male sex. Other pertinent risk factors included older age, Caucasian ethnicity, and a primary diagnosis of proximal humerus fracture. The rate of acute MI in patients who were older than 75 years when they underwent HSA for proximal humerus fracture was 0.80%.

Demographics

We found that patients who had an acute MI after shoulder arthroplasty were likely older, male, and Caucasian. Age and male sex are well-established risk factors for increased cardiac complications after arthroplasty.^27-29 Previous studies have indicated that the rate of cardiac events increases in arthroplasty patients older than 65 years.^19,28,29 In our study, more than 50% of the patients who had an acute perioperative MI were older than 85 years. Less explainable is the increased occurrence of acute MI in Caucasian patients and wealthy patients, given that minorities in the United States have higher rates of cardiovascular disease.³⁰ Shoulder arthroplasty is an elective procedure, more likely to be undertaken by Caucasians. Therefore, at-risk minority groups and financially challenged groups may be less likely to have this procedure.

Primary Diagnosis

In this series, patients with a primary diagnosis of proximal humerus fracture were more likely to have an in-hospital MI. This finding is consistent with previous studies indicating a higher rate of complications for proximal humerus fracture patients than for shoulder arthroplasty patients.^31,32 Given that more than 75% of patients who present with a proximal humerus fracture are older than 70 years, it would be prudent to examine operative indications after this diagnosis,³³ particularly as benefit from surgery for fractures has not been definitively demonstrated.^34-37

Comorbidities

Many of the patients in our MI cohort presented with congestive heart failure, angina pectoris, complicated diabetes, renal failure, fluid and electrolyte disorders, or deficiency anemia. This is in keeping with other studies indicating that preexisting cardiovascular morbidity increases the rate of MI after various forms of arthroplasty.^7-11 Patients in our MI cohort were also susceptible to a variety of post-MI perioperative complications, including pulmonary embolism, pneumonia, deep venous thrombosis, cerebrovascular event, acute renal failure, gastrointestinal complication, mechanical ventilation, transfusion, and nonroutine discharge, and their incidence of death was higher. These findings are consistent with reports that postoperative cardiovascular complications increase the degree of morbidity and mortality in arthroplasty patients.^14-16 It is also worth noting that the odds of MI in the perioperative period increase with each subsequent day of care. This is understandable given that patients presenting with numerous comorbidities are at increased risk for perioperative complications³⁸ resulting in hospital readmission.³⁹

The literature indicates that MI occurs as a complication in 0.7% of patients who undergo noncardiac surgery,⁷ though some series have shown it is more prevalent after arthroplasty procedures.^28,40 MI significantly increases the rate of perioperative morbidity and mortality,⁸ and perioperative cardiac morbidity is a leading cause of death after anesthesia and surgery.¹² Furthermore, the most common cause of death after lower extremity arthroplasty is cardiovascular-related.^41,42 In patients who presented for elective hip arthroplasty, cardiorespiratory disease was one of the main risk factors (with older age and male sex) shown to increase perioperative mortality.⁴³

Perioperative cardiovascular complications increase postoperative morbidity and mortality.¹² The rate of cardiovascular complications after shoulder arthroplasty ranges from 0.8% to 2.6%, and the incidence of MI hovers between 0.3% and 0.9%.^{17,19,28,40,44} A recent study in 793 patients found that, over a 30-day period, cardiovascular complications accounted for more than one-fourth of all complications.¹⁷ Singh and colleagues¹⁹ analyzed cardiopulmonary complications after primary shoulder arthroplasty in a total of 3480 patients (4019 arthroplasties) and found this group had a 90-day cardiac morbidity (MI, congestive heart failure, arrhythmia) rate of 2.6%. In that study, a Deyo-Charlson index of 1 or more was a significant independent risk factor for cardiac complications following surgery. Scores on this weighted index of 17 comorbidities are used to assess the complexities of a patient population. Given the severity of cardiovascular perioperative complications, it is important to preoperatively identify high-risk population groups and sufficiently study and optimize patients before shoulder arthroplasty.

There is much debate about the effectiveness of perioperative β-blockers in reducing perioperative cardiac morbidity and mortality.^45-48 Such a discussion is outside of the scope of this article, but it may be prudent to seek a cardiology consultation for patients presenting with risk factors for perioperative MI. β-Blockers may prove useful in reducing cardiac morbidity in high-risk patients after noncardiac surgery.^45,49

Many limitations are inherent in studies that use a nationally represented database such as NIS, which we used in this study. It is highly likely that NIS does not capture all potential postoperative complications, as this database is very large and subject to errors in data entry and clinical coding. In addition, detailed clinical information (eg, severity of certain comorbid diseases before shoulder arthroplasty, details about the intraoperative course) was not readily available for analysis. Another limitation, which may have led to an underestimate of complication rates, was our not being able to obtain information about postdischarge complications.

Despite these limitations, NIS and other databases have helped researchers answer questions about low-incidence conditions and generalize findings to a national population. In the present study, we analyzed 2 cohorts, patients with and without acute MI after shoulder arthroplasty, to determine predictors for and complications of postarthroplasty MI. We identified numerous predictors for acute MI: congestive heart failure, angina pectoris, complicated diabetes, renal failure, fluid and electrolyte disorders, and deficiency anemia prior to arthroplasty. As perioperative MI is associated with significant morbidity,^14-16 it would be wise to screen patients for such comorbid conditions, assess the severity of these conditions, and offer shoulder arthroplasty with prudence.

Conclusion

The top 4 predictors for acute MI after shoulder arthroplasty were congestive heart failure, angina pectoralis, complicated diabetes mellitus, and male sex. Other pertinent risk factors included older age, Caucasian ethnicity, and primary diagnosis of proximal humerus fracture. Surgeons and patients must be aware of predictors for adverse surgical outcomes such as perioperative MI and understand the extent to which these events increase perioperative morbidity and mortality.

The incidence of shoulder arthroplasty in the United States is increasing annually,^1-3 and the majority of these operations occur in older patients.^4-6 Elderly patients with cardiovascular, pulmonary, cerebral, renal, and hepatic disease are increasingly susceptible to numerous surgical complications.⁴ Myocardial infarction (MI) is a complication that occurs in 0.7% of noncardiac surgeries. This figure increases to 1.1% in patients with coronary artery disease.^7-11 Perioperative MI increases morbidity and mortality,⁸ and perioperative cardiac morbidity is the leading cause of death after anesthesia and surgery.¹² The financial effects of perioperative cardiac morbidity and mortality must also be considered. A 2009 claims analysis study estimated charges associated with a perioperative MI at $15,000 and the cost of cardiac death at $21,909.¹³

Cardiovascular complications are associated with a significant degree of morbidity and mortality in patients who undergo arthroplasty.^14-16 Although studies have elucidated 30- and 90-day morbidity and mortality rates after shoulder arthroplasty, in hip and knee arthroplasty^17-19 little has been done to determine predictors of perioperative MI in a representative database of patients. Given the increasing incidence of shoulder arthroplasty in the United States, the elective nature of this procedure, and the percentage of the US population with cardiovascular risk factors,²⁰ it is important to establish predictors of perioperative MI to ensure patients and physicians have the necessary resources to make informed decisions.

We conducted a study to examine the risk factors for perioperative MI in a large cohort of patients admitted for shoulder arthroplasty to US hospitals. We wanted to evaluate the association between perioperative MI and shoulder arthroplasty with respect to demographics, primary diagnosis, medical comorbidities, and perioperative complications. Specifically, we tested the null hypothesis that, among patients undergoing shoulder arthroplasty, and accounting for confounding variables, there would be no difference in risk factors for patients who have a perioperative MI.

Materials and Methods

This study was exempt from approval by our institutional review board. All data used in this project were deidentified before use.

Nationwide Inpatient Sample (NIS)

The Nationwide Inpatient Sample (NIS), an annual survey of hospitals, is conducted by the Healthcare Cost and Utilization Project (HCUP) and sponsored by the Agency for Healthcare Research and Quality (AHRQ). This database is the largest publicly available all-payer inpatient discharge database in the United States.²¹ Sampling 8 million hospital stays each year, NIS includes information from a representative batch of 20% of US hospitals. In 2011, 46 states and 1045 hospitals contributed information to the database, representing 97% of the US population.²² This large sample allows researchers to analyze a robust set of medical conditions and uncommon treatments. The survey, conducted each year since 1988, includes demographic, clinical, and resource use data.²³ Discharge weight files are provided by NIS to arrive at valid national estimates.

This database is particularly useful because it provides information on up to 25 medical diagnoses and 15 procedures, which are recorded with International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. Researchers can use this database to analyze patient and hospital characteristics as well as inpatient outcomes.^24,25 Numerous studies have used NIS to address pertinent queries across the medical landscape.^22,26

Patient Selection and Analysis

We used NIS to isolate a population of 422,371 adults (≥18 years old) who underwent total shoulder arthroplasty (TSA) or hemi–shoulder arthroplasty (HSA) between January 1, 2002 and December 31, 2011. We then placed the patients in this population into 1 of 2 cohorts. The first cohort had an acute MI during the perioperative period after TSA, and the second, larger cohort did not have an acute MI after TSA. Acute MI was identified using ICD-9-CM code 410.xx. To identify a population of shoulder arthroplasty patients, we included discharges with an ICD-9-CM procedure code of 81.80 or 81.88 (both TSA) or 81.81 (HSA) in the sample. We then considered the degree to which each of 5 variables—primary diagnosis, age, sex, race, and select medical comorbidities—was predictive of in-hospital MI after TSA.

Statistical Analysis

Given the large sample used in this study, normal distribution of data was assumed. Using bivariate analysis, Pearson χ² test for categorical data, and independent-samples t test for continuous data, we compared the nonacute MI and acute MI groups. Multivariable binary logistic regression analyses allowed us to isolate the extent that primary diagnosis, age, sex, race, and medical comorbidities were predictors of acute MI after shoulder arthroplasty. Statistical significance was set at P < .05. SPSS Version 22.0 (SPSS, Chicago, Illinois) was used for all statistical analyses and data modeling.

Results

Between January 1, 2002 and December 31, 2011, an estimated total of 422,371 patients underwent shoulder arthroplasty (59.3% TSA, 40.7% HSA). Of these patients, 1174 (0.28%) had a perioperative MI, and 421,197 (99.72%) did not (Table 1). Patients with a primary diagnosis of proximal humerus fracture (33.8% vs 16.6%; P < .001) or rotator cuff arthropathy (10.1% vs 9.9%; P < .001) were more likely than patients with other diagnoses to have an in-hospital MI.

Our review of the demographics found that patients who underwent shoulder arthroplasty and had a perioperative MI were likely older (75±8.9 years vs 69±11 years; P < .001), Caucasian (94.2% vs 91.9%; P = .002), male (43.2% vs 39.7%; P = .013), in the highest median household income bracket of $63,000 or more (30.8% vs 25.6%; P < .001), and using Medicare (80.9% vs 66.3%; P < .001). They were more likely to be treated in a medical center of medium size (25.6% vs 23.7%; P = .042) or larger (61.8% vs 61.2%; P = .042). MIs occurred more often in urban environments (91.4% vs 88.5%; P = .002) and in HSA patients (55% vs 40.6%; P < .001), resulting in longer hospital  stays (9.4±7.9 days vs 2.7±2.5 days; P < .001) and higher probability of death (6.5% vs 0.1%; P < .001).

We then analyzed the 2 cohorts for medical comorbidities (Table 2). Patients in the MI cohort presented with a significantly higher incidence of congestive heart failure, previous MI, angina pectoris, chronic lung disease, hypertension, diabetes, renal failure, fluid and electrolyte disorders, pulmonary circulatory disease, coagulopathy, and deficiency anemia (P < .001) but not liver disease and obesity. Bivariate analysis of perioperative outcomes (Table 3) indicated that these patients also had a statistically higher rate of numerous other complications: pulmonary embolism (4.9% vs 0.2%; P < .001), pneumonia (15.1% vs 1.2%; P < .001), deep venous thrombosis (2.6% vs 0.2%; P < .001), cerebrovascular event (1.6% vs 0.1%; P < .001), acute renal failure (15.1% vs 1.2%; P < .001), gastrointestinal complication (1.2% vs 0.3%; P < .001), mechanical ventilation (1.2% vs 0.3%; P < .001), transfusion (33.4% vs 8.8%; P < .001), and nonroutine discharge (73.3% vs 36.0%; P < .001).

Multivariable logistic regression analysis was performed to determine independent predictors of perioperative MI after shoulder arthroplasty (Table 4). Patients with a primary diagnosis of proximal humerus fracture (odds ratio [OR], 1.38; 95% confidence interval [CI], 1.15-1.65; P < .001) were more likely than patients with a primary diagnosis of osteoarthritis to have an MI. The odds of postoperative MI increased with age (OR, 1.04 per year; 95% CI, 1.03-1.05; P < .001) and were higher in males (OR, 1.72; 95% CI, 1.52-1.96; P < .001). Compared with Caucasians, African Americans (OR, 0.19; 95% CI, 0.09-0.40; P < .001) were less likely to have an in-hospital MI after shoulder arthroplasty. After shoulder arthroplasty, the odds of MI in the perioperative period increased with each subsequent day of care (OR, 1.10; 95% CI, 1.10-1.11; P < .001).

Regarding independent comorbidities, multivariable logistic regression analysis also determined that history of congestive heart failure (OR, 4.86; 95% CI, 4.20-5.61; P < .001), angina pectoris (OR, 2.90; 95% CI, 2.02-4.17; P < .001), complicated diabetes (OR, 1.96; 95% CI, 1.49-2.57; P < .001), renal failure (OR, 1.42; 95% CI, 1.17-1.72; P < .001), fluid and electrolyte disorders (OR, 1.42; 95% CI, 1.21-1.67; P < .001), and deficiency anemia (OR, 1.62; 95% CI, 1.40-1.88; P < .001) were significant predictors of perioperative MI after shoulder arthroplasty.

Discussion

Results of other studies have elucidated 30- and 90-day mortality rates and postoperative complications after shoulder arthroplasty, but, relative to hip and knee arthroplasty,^17-19 little has been done to determine predictors of perioperative MI in a large sample of shoulder arthroplasty patients. Given the increasing rates of shoulder arthroplasty^1-3 and the demographics of this population,^4-6 it is likely that postoperative cardiovascular events will increase in frequency. We found that, in order of decreasing significance, the top 4 risk predictors for acute MI after shoulder arthroplasty were congestive heart failure, angina pectoralis, complicated diabetes mellitus, and male sex. Other pertinent risk factors included older age, Caucasian ethnicity, and a primary diagnosis of proximal humerus fracture. The rate of acute MI in patients who were older than 75 years when they underwent HSA for proximal humerus fracture was 0.80%.

Demographics

We found that patients who had an acute MI after shoulder arthroplasty were likely older, male, and Caucasian. Age and male sex are well-established risk factors for increased cardiac complications after arthroplasty.^27-29 Previous studies have indicated that the rate of cardiac events increases in arthroplasty patients older than 65 years.^19,28,29 In our study, more than 50% of the patients who had an acute perioperative MI were older than 85 years. Less explainable is the increased occurrence of acute MI in Caucasian patients and wealthy patients, given that minorities in the United States have higher rates of cardiovascular disease.³⁰ Shoulder arthroplasty is an elective procedure, more likely to be undertaken by Caucasians. Therefore, at-risk minority groups and financially challenged groups may be less likely to have this procedure.

Primary Diagnosis

In this series, patients with a primary diagnosis of proximal humerus fracture were more likely to have an in-hospital MI. This finding is consistent with previous studies indicating a higher rate of complications for proximal humerus fracture patients than for shoulder arthroplasty patients.^31,32 Given that more than 75% of patients who present with a proximal humerus fracture are older than 70 years, it would be prudent to examine operative indications after this diagnosis,³³ particularly as benefit from surgery for fractures has not been definitively demonstrated.^34-37

Comorbidities

Many of the patients in our MI cohort presented with congestive heart failure, angina pectoris, complicated diabetes, renal failure, fluid and electrolyte disorders, or deficiency anemia. This is in keeping with other studies indicating that preexisting cardiovascular morbidity increases the rate of MI after various forms of arthroplasty.^7-11 Patients in our MI cohort were also susceptible to a variety of post-MI perioperative complications, including pulmonary embolism, pneumonia, deep venous thrombosis, cerebrovascular event, acute renal failure, gastrointestinal complication, mechanical ventilation, transfusion, and nonroutine discharge, and their incidence of death was higher. These findings are consistent with reports that postoperative cardiovascular complications increase the degree of morbidity and mortality in arthroplasty patients.^14-16 It is also worth noting that the odds of MI in the perioperative period increase with each subsequent day of care. This is understandable given that patients presenting with numerous comorbidities are at increased risk for perioperative complications³⁸ resulting in hospital readmission.³⁹

The literature indicates that MI occurs as a complication in 0.7% of patients who undergo noncardiac surgery,⁷ though some series have shown it is more prevalent after arthroplasty procedures.^28,40 MI significantly increases the rate of perioperative morbidity and mortality,⁸ and perioperative cardiac morbidity is a leading cause of death after anesthesia and surgery.¹² Furthermore, the most common cause of death after lower extremity arthroplasty is cardiovascular-related.^41,42 In patients who presented for elective hip arthroplasty, cardiorespiratory disease was one of the main risk factors (with older age and male sex) shown to increase perioperative mortality.⁴³

Perioperative cardiovascular complications increase postoperative morbidity and mortality.¹² The rate of cardiovascular complications after shoulder arthroplasty ranges from 0.8% to 2.6%, and the incidence of MI hovers between 0.3% and 0.9%.^{17,19,28,40,44} A recent study in 793 patients found that, over a 30-day period, cardiovascular complications accounted for more than one-fourth of all complications.¹⁷ Singh and colleagues¹⁹ analyzed cardiopulmonary complications after primary shoulder arthroplasty in a total of 3480 patients (4019 arthroplasties) and found this group had a 90-day cardiac morbidity (MI, congestive heart failure, arrhythmia) rate of 2.6%. In that study, a Deyo-Charlson index of 1 or more was a significant independent risk factor for cardiac complications following surgery. Scores on this weighted index of 17 comorbidities are used to assess the complexities of a patient population. Given the severity of cardiovascular perioperative complications, it is important to preoperatively identify high-risk population groups and sufficiently study and optimize patients before shoulder arthroplasty.

There is much debate about the effectiveness of perioperative β-blockers in reducing perioperative cardiac morbidity and mortality.^45-48 Such a discussion is outside of the scope of this article, but it may be prudent to seek a cardiology consultation for patients presenting with risk factors for perioperative MI. β-Blockers may prove useful in reducing cardiac morbidity in high-risk patients after noncardiac surgery.^45,49

Many limitations are inherent in studies that use a nationally represented database such as NIS, which we used in this study. It is highly likely that NIS does not capture all potential postoperative complications, as this database is very large and subject to errors in data entry and clinical coding. In addition, detailed clinical information (eg, severity of certain comorbid diseases before shoulder arthroplasty, details about the intraoperative course) was not readily available for analysis. Another limitation, which may have led to an underestimate of complication rates, was our not being able to obtain information about postdischarge complications.

Despite these limitations, NIS and other databases have helped researchers answer questions about low-incidence conditions and generalize findings to a national population. In the present study, we analyzed 2 cohorts, patients with and without acute MI after shoulder arthroplasty, to determine predictors for and complications of postarthroplasty MI. We identified numerous predictors for acute MI: congestive heart failure, angina pectoris, complicated diabetes, renal failure, fluid and electrolyte disorders, and deficiency anemia prior to arthroplasty. As perioperative MI is associated with significant morbidity,^14-16 it would be wise to screen patients for such comorbid conditions, assess the severity of these conditions, and offer shoulder arthroplasty with prudence.

Conclusion

The top 4 predictors for acute MI after shoulder arthroplasty were congestive heart failure, angina pectoralis, complicated diabetes mellitus, and male sex. Other pertinent risk factors included older age, Caucasian ethnicity, and primary diagnosis of proximal humerus fracture. Surgeons and patients must be aware of predictors for adverse surgical outcomes such as perioperative MI and understand the extent to which these events increase perioperative morbidity and mortality.

The diagnosis of cancer is a life-changing event for the patient as well as the patient’s family, friends, and relatives. Once diagnosed, most cancer patients want more information about their prognosis, future procedures, and/or treatment options.¹ Receiving such information has been shown to reduce patient anxiety, increase patient satisfaction with care, and improve self-care.^2-6 With the evolution of the Internet, patients in general^7-9 and, specifically, cancer patients^10-17 have turned to websites and online patient education materials (PEMs) to gather more health information.

For online PEMs to convey health information, their reading level must match the health literacy of the individuals who access them. Health literacy is the ability of an individual to gather and comprehend information about their condition to make the best decisions for their health.¹⁸ According to a report by the Institute of Medicine, 90 million American adults cannot properly use the US health care system because they do not possess adequate health literacy.¹⁸ Additionally, 36% of adults in the United States have basic or less-than-basic health literacy.¹⁹ This is starkly contrasted with the 12% of US adults who have proficient health literacy. A 2012 survey showed that about 31% of individuals who look for health information on the Internet have a high school education or less.⁸ In order to address the low health literacy of adults, the National Institutes of Health (NIH) has recommended that online PEMs be written at a sixth- to seventh-grade reading level.²⁰ 

Unfortunately, many online PEMs related to certain cancer^21-25 and orthopedic conditions^26-31 do not meet NIH recommendations. Only 1 study has specifically looked at PEMs related to an orthopedic cancer condition.³² Lam and colleagues³² evaluated the readability of osteosarcoma PEMs from 56 websites using only 2 readability instruments and identified 86% of the websites as having a greater than eighth-grade reading level. No study has thoroughly assessed the readability of PEMs about bone and soft-tissue sarcomas and related conditions nor has any used 10 different readability instruments. Since each readability instrument has different variables (eg, sentence length, number of paragraphs, or number of complex words), averaging the scores of 10 of these instruments may result in less bias.

The purpose of this study was to evaluate the readability of online PEMs concerning bone and soft-tissue sarcomas and related conditions. The online PEMs came from websites that sarcoma patients may visit to obtain information about their condition. Our hypothesis was that the majority of these online PEMs will have a higher reading level than the NIH recommendations.

Materials and Methods

In May 2013, we identified online PEMs that included background, diagnosis, tests, or treatments for bone and soft-tissue sarcomas and conditions that mimic bone sarcoma. We included articles from the Tumors section of the American Academy of Orthopaedic Surgeons (AAOS) website.³³ A second source of online PEMs came from a list of academic training centers created through the American Medical Association’s Fellowship and Residency Electronic Internet Database (FREIDA) with search criteria narrowed to orthopedic surgery. If we did not find PEMs of bone and soft-tissue cancers in the orthopedic department of a given academic training center’s website, we searched its cancer center website. We chose 4 programs with PEMs relevant to bone and soft-tissue sarcomas from each region in FREIDA for a balanced representation, except for the Territory region because it had only 1 academic training center and no relevant PEMs. Specialized websites, including Bonetumor.org, Sarcoma Alliance (Sarcomaalliance.org), and Sarcoma Foundation of America (Curesarcoma.org), were also evaluated. Within the Sarcoma Specialists section of the Sarcoma Alliance website,³⁴ sarcoma specialists who were not identified from the FREIDA search for academic training centers were selected for review.

Because 8 of 10 individuals looking for health information on the Internet start their investigation at search engines, we also looked for PEMs through a Google search (Google.com) of bone cancer, and evaluated the first 10 hits for PEMs.⁸ Of these 10 hits, 8 had relevant PEMs, which we searched for additional PEMs about bone and soft-tissue cancers and related conditions. We also conducted a Google search of the most common bone sarcoma and soft-tissue sarcoma, osteosarcoma and malignant fibrous histiocytoma, respectively, and found 2 additional websites with relevant PEMs. LaCoursiere and colleagues³⁵ surveyed cancer patients who used the Internet and found that they preferred WebMD (Webmd.com) and Medscape (Medscape.com) as sources for content about their medical condition.³⁵ WebMD had been identified in the Google search, and we gathered the PEMs from Medscape also. It is worth noting that some of these websites are written for patients as well as clinicians.

Text from these PEMs were copied and pasted into separate Microsoft Word documents (Microsoft, Redmond, Washington). Advertisements, pictures, picture text, hyperlinks, copyright notices, page navigation links, paragraphs with no text, and any text that was not related to the given condition were deleted from the document to format the text for the readability software. Then, each Microsoft Word document was uploaded into the software package Readability Studio Professional (RSP) Edition Version 2012.1 for Windows (Oleander Software, Vandalia, Ohio). The 10 distinct readability instruments that were used to gauge the readability of each document were the Flesch Reading Ease score (FRE), the New Fog Count, the New Automated Readability Index, the Coleman-Liau Index (CLI), the Fry readability graph, the New Dale-Chall formula (NDC), the Gunning Frequency of Gobbledygook (Gunning FOG), the Powers-Sumner-Kearl formula, the Simple Measure of Gobbledygook (SMOG), and the Raygor Estimate Graph.

The FRE’s formula takes the average number of words per sentence and average number of syllables per word to compute a score ranging from 0 to 100 with 0 being the hardest to read.³⁶ The New Fog Count tallies the number of sentences, easy words, and hard words (polysyllables) to calculate the grade level of the document.³⁷ The New Automated Readability Index takes the average characters per word and average words per sentence to calculate a grade level for the document.³⁷ The CLI randomly samples a few hundred words from the document, averages the number of letters and sentences per sample, and calculates an estimated grade level.³⁸ The Fry readability graph selects samples of 100 words from the document, averages the number of syllables and sentences per 100 words, plots these data points on a graph, with the intersection determining the reading level.³⁹ The NDC uses a list of 3000 familiar words that most fourth-grade students know.⁴⁰ The percentage of difficult words, which are not on the list of familiar words, and the average sentence length in words are used to calculate the reading grade level of the document. The Gunning FOG uses the average sentence length in words and the percentage of hard words from a sample of at least 100 words to determine the reading grade level of the document.⁴¹ The Powers-Sumner-Kearl formula uses the average sentence length and percentage of monosyllables from a 100-word sample passage to calculate the reading grade level.⁴² The SMOG formula counts the number of polysyllabic words from 30 sentences and calculates the reading grade level of the document.⁴³ In contrast to other formulas that test for 50% to 75% comprehension, the SMOG formula tests for 100% comprehension. As a result, the SMOG formula generally assigns a reading level 2 grades higher than the Dale-Chall level. The Raygor Estimate Graph selects a 100-word passage, counts the number of sentences and number of words with 6 or more letters, and plots the 2 variables on a graph to determine the reading grade level.⁴⁴ The software package calculated the results from each reading instrument and reported the mean grade level score
for each document.

Results

We identified a total of 72 websites with relevant PEMs and included them in this study. Of these 72 websites, 36 websites were academic training centers, 10 were Google search hits, and 21 were from the Sarcoma Alliance list of sarcoma specialists. The remaining 5 websites were AAOS, Bonetumor.org, Sarcoma Alliance, Sarcoma Foundation of America, and Medscape. A list of conditions and treatments that were considered relevant PEMs is found in Appendix 1. A total of 774 articles were obtained from the 72 websites.

None of the websites had a mean readability score of 7 (seventh grade) or lower (Figures 1A, 1B). Mid-America Sarcoma Institute’s PEMs had the lowest mean readability score, 8.9. The lowest readability score was 5.3, which the New Fog Count readability instrument calculated for Vanderbilt University Medical Center’s (VUMC’s) PEMs (Appendix 2). The mean readability score of all websites was 11.4 (range, 8.9-15.5) (Appendix 2).

Seventy of 72 websites (97%) had PEMs that were fairly difficult or difficult, according to the FRE analysis (Figure 2). The American Cancer Society and Mid-America Sarcoma Institute had PEMs that were written in plain English. Sixty-nine of 72 websites (96%) had PEMs with a readability score of 10 or higher, according to the Raygor readability estimate (Figure 3). Using this instrument, the scores of the American Cancer Society and the University of Pennsylvania–Joan Karnell Cancer Center were 9; Mid-America Sarcoma Institute’s score was 8. 

Discussion

Many cancer patients have turned to websites and online PEMs to gather health information about their condition.^10-17 Basch and colleagues¹⁰ reported almost a decade ago that 44% of cancer patients, as well as 60% of their companions, used the Internet to find cancer-related information.¹⁰ When LaCoursiere and colleagues³⁵ surveyed cancer patients, they found that patients handled their condition better and had less anxiety and uncertainty after using the Internet to find health information and support.³⁵ In addition, many orthopedic patients, specifically 46% of orthopedic community outpatients,⁴⁵ consult the Internet for information about their condition and future surgical procedures.^46,47

This study comprehensively evaluated the readability of online PEMs of bone and soft-tissue sarcomas and related conditions by using 10 different readability instruments. After identifying 72 websites and 774 articles, we found that all 72 websites’ PEMs had a mean readability score that did not meet the NIH recommendation of writing PEMs at a sixth- to seventh-grade reading level. These results are consistent with studies evaluating the readability of online PEMs related to other cancer conditions^21-25 and other orthopedic conditions.^26-31

The combination of low health literacy of many US adults and high reading grade levels of the majority of online PEMs is not conducive to patients’ better understanding their condition(s). Even individuals with high reading skills prefer information that is simpler to read.⁴⁸ In many areas of medicine, there is evidence that patients’ understanding of their condition has a positive impact on health outcomes, well-being, and the patient–physician relationship.^49-61 Regarding cancer patients, Davis and colleagues⁵⁴ and Peterson and colleagues⁵⁷ showed that lower health literacy contributes to less knowledge and lower rates of breast⁵⁴ and colorectal cancer⁵⁷ screening tests. Even low health literacy of family caregivers of cancer patients can result in increased stress and lack of communication of important medical information between caregiver and physician.⁵² Among cancer patients, poor health literacy has been associated with mental distress⁶⁰ as well as decreased compliance with treatment and lower involvement in clinical trials.⁵⁵

The disparity between patients’ health literacy and the readability of online PEMs needs to be addressed by finding methods to improve patients’ understanding of their condition and to lower the readability scores of online PEMs. Better communication between patient and physician may improve patients’ comprehension of their condition and different aspects of their care.^59,62-66 Doak and colleagues⁶³ recommend giving cancer patients the most important information first; presenting information to patients in smaller doses; intermittently asking patients questions; and incorporating graphs, tables, and drawings into communication with patients.⁶³ Additionally, allowing patients to repeat information they have just received/heard to the physician is another useful tool to improve patient education.^62,64-66

Another way to address the disparity between patients’ health literacy and the readability of online PEMs is to reduce the reading grade level of existing PEMs. According to results from this study and others, the majority of online PEMs are above the reading grade level of a significant number of US adults. Many available and inexpensive readability instruments allow authors to assess their articles’ readability. Many writing guidelines also exist to help authors improve the readability of their PEMs.^20,64,67-71 Living Word Vocabulary⁷⁰ and Plain Language⁷¹ help authors replace complex words or medical terms with simpler words.²⁹ Visual aids, audio, and video help patients with low health literacy remember the information.⁶⁴

Efforts to improve PEM readability are effective. Of all the websites reviewed, VUMC was identified as having PEMs with the lowest readability score (5.3). This score was reported by the New Fog Count readability instrument, which accounts for the number of sentences, easy words, and hard words. In 2011, VUMC formed the Department of Patient Education to review and update its online and printed PEMs to make sure patients could read them.⁷² Additionally, the mean readability scores of the websites of the National Cancer Institute and MedlinePlus are in the top 50% of the websites included in this study. The NIH sponsors both sites, which follow the NIH guidelines for writing online PEMs at a reading level suitable for individuals with lower health literacy.²⁰ These materials serve as potential models to improve the readability of PEMs, and, thus, help patients to better understand their condition, medical procedures, and/or treatment options.

To illustrate ways to improve the reading grade level of PEMs, we used the article “Ewing’s Sarcoma” from the AAOS website⁷³ and followed the NIH guidelines to improve the reading grade level of the article.²⁰ We identified complex words and defined them at an eighth-grade reading level. If that word was mentioned later in the article, simpler terminology was used instead of the initial complex word. For example, Ewing’s sarcoma was defined early and then referred to as bone tumor later in the article. We also identified every word that was 3 syllables or longer and used Microsoft Word’s thesaurus to replace those words with ones that were less than 3 syllables. Lastly, all sentences longer than 15 words were rewritten to be less than 15 words. After making these 3 changes to the article, the mean reading grade level dropped from 11.2 to 7.3. 

This study has limitations. First, some readability instruments evaluate the number of syllables per word or polysyllabic words as part of their formula and, thus, can underestimate or overestimate the reading grade level of a document. Some readability formulas consider medical terms such as ulna, femur, or carpal as “easy” words because they have 2 syllables, but many laypersons may not comprehend these words. On the other hand, some readability formulas consider medical terms such as medications, diagnosis, or radiation as “hard” words because they contain 3 or more syllables, but the majority of laypersons likely comprehend these words. Second, the reading level of the patient population accessing those online sites was not assessed. Third, the readability instruments in this study did not evaluate the accuracy of the content, pictures, or tables of the PEMs. However, using 10 readability instruments allowed evaluation of many different readability aspects of the text. Fourth, because some websites identified in this study, such as Bonetumor.org, were written for patients as well as clinicians, the reading grade level of these sites may be higher than that of those sites written just for patients.

Conclusion

Because many orthopedic cancer patients rely on the Internet as a source of information, the need for online PEMs to match the reading skills of the patient population who accesses them is vital. However, this study shows that many organizations, academic training centers, and other entities need to update their online PEMs because all PEMs in this study had a mean readability grade level higher than the NIH recommendation. Further research needs to evaluate the effectiveness of other media, such as video, illustrations, and audio, to provide health information to patients. With many guidelines available that provide plans and advice to improve the readability of PEMs, research also must assess the most effective plans and advice in order to allow authors to focus their attention on 1 set of guidelines to improve the readability of their PEMs.

The diagnosis of cancer is a life-changing event for the patient as well as the patient’s family, friends, and relatives. Once diagnosed, most cancer patients want more information about their prognosis, future procedures, and/or treatment options.¹ Receiving such information has been shown to reduce patient anxiety, increase patient satisfaction with care, and improve self-care.^2-6 With the evolution of the Internet, patients in general^7-9 and, specifically, cancer patients^10-17 have turned to websites and online patient education materials (PEMs) to gather more health information.

For online PEMs to convey health information, their reading level must match the health literacy of the individuals who access them. Health literacy is the ability of an individual to gather and comprehend information about their condition to make the best decisions for their health.¹⁸ According to a report by the Institute of Medicine, 90 million American adults cannot properly use the US health care system because they do not possess adequate health literacy.¹⁸ Additionally, 36% of adults in the United States have basic or less-than-basic health literacy.¹⁹ This is starkly contrasted with the 12% of US adults who have proficient health literacy. A 2012 survey showed that about 31% of individuals who look for health information on the Internet have a high school education or less.⁸ In order to address the low health literacy of adults, the National Institutes of Health (NIH) has recommended that online PEMs be written at a sixth- to seventh-grade reading level.²⁰ 

Unfortunately, many online PEMs related to certain cancer^21-25 and orthopedic conditions^26-31 do not meet NIH recommendations. Only 1 study has specifically looked at PEMs related to an orthopedic cancer condition.³² Lam and colleagues³² evaluated the readability of osteosarcoma PEMs from 56 websites using only 2 readability instruments and identified 86% of the websites as having a greater than eighth-grade reading level. No study has thoroughly assessed the readability of PEMs about bone and soft-tissue sarcomas and related conditions nor has any used 10 different readability instruments. Since each readability instrument has different variables (eg, sentence length, number of paragraphs, or number of complex words), averaging the scores of 10 of these instruments may result in less bias.

The purpose of this study was to evaluate the readability of online PEMs concerning bone and soft-tissue sarcomas and related conditions. The online PEMs came from websites that sarcoma patients may visit to obtain information about their condition. Our hypothesis was that the majority of these online PEMs will have a higher reading level than the NIH recommendations.

Materials and Methods

In May 2013, we identified online PEMs that included background, diagnosis, tests, or treatments for bone and soft-tissue sarcomas and conditions that mimic bone sarcoma. We included articles from the Tumors section of the American Academy of Orthopaedic Surgeons (AAOS) website.³³ A second source of online PEMs came from a list of academic training centers created through the American Medical Association’s Fellowship and Residency Electronic Internet Database (FREIDA) with search criteria narrowed to orthopedic surgery. If we did not find PEMs of bone and soft-tissue cancers in the orthopedic department of a given academic training center’s website, we searched its cancer center website. We chose 4 programs with PEMs relevant to bone and soft-tissue sarcomas from each region in FREIDA for a balanced representation, except for the Territory region because it had only 1 academic training center and no relevant PEMs. Specialized websites, including Bonetumor.org, Sarcoma Alliance (Sarcomaalliance.org), and Sarcoma Foundation of America (Curesarcoma.org), were also evaluated. Within the Sarcoma Specialists section of the Sarcoma Alliance website,³⁴ sarcoma specialists who were not identified from the FREIDA search for academic training centers were selected for review.

Because 8 of 10 individuals looking for health information on the Internet start their investigation at search engines, we also looked for PEMs through a Google search (Google.com) of bone cancer, and evaluated the first 10 hits for PEMs.⁸ Of these 10 hits, 8 had relevant PEMs, which we searched for additional PEMs about bone and soft-tissue cancers and related conditions. We also conducted a Google search of the most common bone sarcoma and soft-tissue sarcoma, osteosarcoma and malignant fibrous histiocytoma, respectively, and found 2 additional websites with relevant PEMs. LaCoursiere and colleagues³⁵ surveyed cancer patients who used the Internet and found that they preferred WebMD (Webmd.com) and Medscape (Medscape.com) as sources for content about their medical condition.³⁵ WebMD had been identified in the Google search, and we gathered the PEMs from Medscape also. It is worth noting that some of these websites are written for patients as well as clinicians.

Text from these PEMs were copied and pasted into separate Microsoft Word documents (Microsoft, Redmond, Washington). Advertisements, pictures, picture text, hyperlinks, copyright notices, page navigation links, paragraphs with no text, and any text that was not related to the given condition were deleted from the document to format the text for the readability software. Then, each Microsoft Word document was uploaded into the software package Readability Studio Professional (RSP) Edition Version 2012.1 for Windows (Oleander Software, Vandalia, Ohio). The 10 distinct readability instruments that were used to gauge the readability of each document were the Flesch Reading Ease score (FRE), the New Fog Count, the New Automated Readability Index, the Coleman-Liau Index (CLI), the Fry readability graph, the New Dale-Chall formula (NDC), the Gunning Frequency of Gobbledygook (Gunning FOG), the Powers-Sumner-Kearl formula, the Simple Measure of Gobbledygook (SMOG), and the Raygor Estimate Graph.

The FRE’s formula takes the average number of words per sentence and average number of syllables per word to compute a score ranging from 0 to 100 with 0 being the hardest to read.³⁶ The New Fog Count tallies the number of sentences, easy words, and hard words (polysyllables) to calculate the grade level of the document.³⁷ The New Automated Readability Index takes the average characters per word and average words per sentence to calculate a grade level for the document.³⁷ The CLI randomly samples a few hundred words from the document, averages the number of letters and sentences per sample, and calculates an estimated grade level.³⁸ The Fry readability graph selects samples of 100 words from the document, averages the number of syllables and sentences per 100 words, plots these data points on a graph, with the intersection determining the reading level.³⁹ The NDC uses a list of 3000 familiar words that most fourth-grade students know.⁴⁰ The percentage of difficult words, which are not on the list of familiar words, and the average sentence length in words are used to calculate the reading grade level of the document. The Gunning FOG uses the average sentence length in words and the percentage of hard words from a sample of at least 100 words to determine the reading grade level of the document.⁴¹ The Powers-Sumner-Kearl formula uses the average sentence length and percentage of monosyllables from a 100-word sample passage to calculate the reading grade level.⁴² The SMOG formula counts the number of polysyllabic words from 30 sentences and calculates the reading grade level of the document.⁴³ In contrast to other formulas that test for 50% to 75% comprehension, the SMOG formula tests for 100% comprehension. As a result, the SMOG formula generally assigns a reading level 2 grades higher than the Dale-Chall level. The Raygor Estimate Graph selects a 100-word passage, counts the number of sentences and number of words with 6 or more letters, and plots the 2 variables on a graph to determine the reading grade level.⁴⁴ The software package calculated the results from each reading instrument and reported the mean grade level score
for each document.

Results

We identified a total of 72 websites with relevant PEMs and included them in this study. Of these 72 websites, 36 websites were academic training centers, 10 were Google search hits, and 21 were from the Sarcoma Alliance list of sarcoma specialists. The remaining 5 websites were AAOS, Bonetumor.org, Sarcoma Alliance, Sarcoma Foundation of America, and Medscape. A list of conditions and treatments that were considered relevant PEMs is found in Appendix 1. A total of 774 articles were obtained from the 72 websites.

None of the websites had a mean readability score of 7 (seventh grade) or lower (Figures 1A, 1B). Mid-America Sarcoma Institute’s PEMs had the lowest mean readability score, 8.9. The lowest readability score was 5.3, which the New Fog Count readability instrument calculated for Vanderbilt University Medical Center’s (VUMC’s) PEMs (Appendix 2). The mean readability score of all websites was 11.4 (range, 8.9-15.5) (Appendix 2).

Seventy of 72 websites (97%) had PEMs that were fairly difficult or difficult, according to the FRE analysis (Figure 2). The American Cancer Society and Mid-America Sarcoma Institute had PEMs that were written in plain English. Sixty-nine of 72 websites (96%) had PEMs with a readability score of 10 or higher, according to the Raygor readability estimate (Figure 3). Using this instrument, the scores of the American Cancer Society and the University of Pennsylvania–Joan Karnell Cancer Center were 9; Mid-America Sarcoma Institute’s score was 8. 

Discussion

Many cancer patients have turned to websites and online PEMs to gather health information about their condition.^10-17 Basch and colleagues¹⁰ reported almost a decade ago that 44% of cancer patients, as well as 60% of their companions, used the Internet to find cancer-related information.¹⁰ When LaCoursiere and colleagues³⁵ surveyed cancer patients, they found that patients handled their condition better and had less anxiety and uncertainty after using the Internet to find health information and support.³⁵ In addition, many orthopedic patients, specifically 46% of orthopedic community outpatients,⁴⁵ consult the Internet for information about their condition and future surgical procedures.^46,47

This study comprehensively evaluated the readability of online PEMs of bone and soft-tissue sarcomas and related conditions by using 10 different readability instruments. After identifying 72 websites and 774 articles, we found that all 72 websites’ PEMs had a mean readability score that did not meet the NIH recommendation of writing PEMs at a sixth- to seventh-grade reading level. These results are consistent with studies evaluating the readability of online PEMs related to other cancer conditions^21-25 and other orthopedic conditions.^26-31

The combination of low health literacy of many US adults and high reading grade levels of the majority of online PEMs is not conducive to patients’ better understanding their condition(s). Even individuals with high reading skills prefer information that is simpler to read.⁴⁸ In many areas of medicine, there is evidence that patients’ understanding of their condition has a positive impact on health outcomes, well-being, and the patient–physician relationship.^49-61 Regarding cancer patients, Davis and colleagues⁵⁴ and Peterson and colleagues⁵⁷ showed that lower health literacy contributes to less knowledge and lower rates of breast⁵⁴ and colorectal cancer⁵⁷ screening tests. Even low health literacy of family caregivers of cancer patients can result in increased stress and lack of communication of important medical information between caregiver and physician.⁵² Among cancer patients, poor health literacy has been associated with mental distress⁶⁰ as well as decreased compliance with treatment and lower involvement in clinical trials.⁵⁵

The disparity between patients’ health literacy and the readability of online PEMs needs to be addressed by finding methods to improve patients’ understanding of their condition and to lower the readability scores of online PEMs. Better communication between patient and physician may improve patients’ comprehension of their condition and different aspects of their care.^59,62-66 Doak and colleagues⁶³ recommend giving cancer patients the most important information first; presenting information to patients in smaller doses; intermittently asking patients questions; and incorporating graphs, tables, and drawings into communication with patients.⁶³ Additionally, allowing patients to repeat information they have just received/heard to the physician is another useful tool to improve patient education.^62,64-66

Another way to address the disparity between patients’ health literacy and the readability of online PEMs is to reduce the reading grade level of existing PEMs. According to results from this study and others, the majority of online PEMs are above the reading grade level of a significant number of US adults. Many available and inexpensive readability instruments allow authors to assess their articles’ readability. Many writing guidelines also exist to help authors improve the readability of their PEMs.^20,64,67-71 Living Word Vocabulary⁷⁰ and Plain Language⁷¹ help authors replace complex words or medical terms with simpler words.²⁹ Visual aids, audio, and video help patients with low health literacy remember the information.⁶⁴

Efforts to improve PEM readability are effective. Of all the websites reviewed, VUMC was identified as having PEMs with the lowest readability score (5.3). This score was reported by the New Fog Count readability instrument, which accounts for the number of sentences, easy words, and hard words. In 2011, VUMC formed the Department of Patient Education to review and update its online and printed PEMs to make sure patients could read them.⁷² Additionally, the mean readability scores of the websites of the National Cancer Institute and MedlinePlus are in the top 50% of the websites included in this study. The NIH sponsors both sites, which follow the NIH guidelines for writing online PEMs at a reading level suitable for individuals with lower health literacy.²⁰ These materials serve as potential models to improve the readability of PEMs, and, thus, help patients to better understand their condition, medical procedures, and/or treatment options.

To illustrate ways to improve the reading grade level of PEMs, we used the article “Ewing’s Sarcoma” from the AAOS website⁷³ and followed the NIH guidelines to improve the reading grade level of the article.²⁰ We identified complex words and defined them at an eighth-grade reading level. If that word was mentioned later in the article, simpler terminology was used instead of the initial complex word. For example, Ewing’s sarcoma was defined early and then referred to as bone tumor later in the article. We also identified every word that was 3 syllables or longer and used Microsoft Word’s thesaurus to replace those words with ones that were less than 3 syllables. Lastly, all sentences longer than 15 words were rewritten to be less than 15 words. After making these 3 changes to the article, the mean reading grade level dropped from 11.2 to 7.3. 

This study has limitations. First, some readability instruments evaluate the number of syllables per word or polysyllabic words as part of their formula and, thus, can underestimate or overestimate the reading grade level of a document. Some readability formulas consider medical terms such as ulna, femur, or carpal as “easy” words because they have 2 syllables, but many laypersons may not comprehend these words. On the other hand, some readability formulas consider medical terms such as medications, diagnosis, or radiation as “hard” words because they contain 3 or more syllables, but the majority of laypersons likely comprehend these words. Second, the reading level of the patient population accessing those online sites was not assessed. Third, the readability instruments in this study did not evaluate the accuracy of the content, pictures, or tables of the PEMs. However, using 10 readability instruments allowed evaluation of many different readability aspects of the text. Fourth, because some websites identified in this study, such as Bonetumor.org, were written for patients as well as clinicians, the reading grade level of these sites may be higher than that of those sites written just for patients.

Conclusion

Because many orthopedic cancer patients rely on the Internet as a source of information, the need for online PEMs to match the reading skills of the patient population who accesses them is vital. However, this study shows that many organizations, academic training centers, and other entities need to update their online PEMs because all PEMs in this study had a mean readability grade level higher than the NIH recommendation. Further research needs to evaluate the effectiveness of other media, such as video, illustrations, and audio, to provide health information to patients. With many guidelines available that provide plans and advice to improve the readability of PEMs, research also must assess the most effective plans and advice in order to allow authors to focus their attention on 1 set of guidelines to improve the readability of their PEMs.

The diagnosis of cancer is a life-changing event for the patient as well as the patient’s family, friends, and relatives. Once diagnosed, most cancer patients want more information about their prognosis, future procedures, and/or treatment options.¹ Receiving such information has been shown to reduce patient anxiety, increase patient satisfaction with care, and improve self-care.^2-6 With the evolution of the Internet, patients in general^7-9 and, specifically, cancer patients^10-17 have turned to websites and online patient education materials (PEMs) to gather more health information.

For online PEMs to convey health information, their reading level must match the health literacy of the individuals who access them. Health literacy is the ability of an individual to gather and comprehend information about their condition to make the best decisions for their health.¹⁸ According to a report by the Institute of Medicine, 90 million American adults cannot properly use the US health care system because they do not possess adequate health literacy.¹⁸ Additionally, 36% of adults in the United States have basic or less-than-basic health literacy.¹⁹ This is starkly contrasted with the 12% of US adults who have proficient health literacy. A 2012 survey showed that about 31% of individuals who look for health information on the Internet have a high school education or less.⁸ In order to address the low health literacy of adults, the National Institutes of Health (NIH) has recommended that online PEMs be written at a sixth- to seventh-grade reading level.²⁰ 

Unfortunately, many online PEMs related to certain cancer^21-25 and orthopedic conditions^26-31 do not meet NIH recommendations. Only 1 study has specifically looked at PEMs related to an orthopedic cancer condition.³² Lam and colleagues³² evaluated the readability of osteosarcoma PEMs from 56 websites using only 2 readability instruments and identified 86% of the websites as having a greater than eighth-grade reading level. No study has thoroughly assessed the readability of PEMs about bone and soft-tissue sarcomas and related conditions nor has any used 10 different readability instruments. Since each readability instrument has different variables (eg, sentence length, number of paragraphs, or number of complex words), averaging the scores of 10 of these instruments may result in less bias.

The purpose of this study was to evaluate the readability of online PEMs concerning bone and soft-tissue sarcomas and related conditions. The online PEMs came from websites that sarcoma patients may visit to obtain information about their condition. Our hypothesis was that the majority of these online PEMs will have a higher reading level than the NIH recommendations.

Materials and Methods

In May 2013, we identified online PEMs that included background, diagnosis, tests, or treatments for bone and soft-tissue sarcomas and conditions that mimic bone sarcoma. We included articles from the Tumors section of the American Academy of Orthopaedic Surgeons (AAOS) website.³³ A second source of online PEMs came from a list of academic training centers created through the American Medical Association’s Fellowship and Residency Electronic Internet Database (FREIDA) with search criteria narrowed to orthopedic surgery. If we did not find PEMs of bone and soft-tissue cancers in the orthopedic department of a given academic training center’s website, we searched its cancer center website. We chose 4 programs with PEMs relevant to bone and soft-tissue sarcomas from each region in FREIDA for a balanced representation, except for the Territory region because it had only 1 academic training center and no relevant PEMs. Specialized websites, including Bonetumor.org, Sarcoma Alliance (Sarcomaalliance.org), and Sarcoma Foundation of America (Curesarcoma.org), were also evaluated. Within the Sarcoma Specialists section of the Sarcoma Alliance website,³⁴ sarcoma specialists who were not identified from the FREIDA search for academic training centers were selected for review.

Because 8 of 10 individuals looking for health information on the Internet start their investigation at search engines, we also looked for PEMs through a Google search (Google.com) of bone cancer, and evaluated the first 10 hits for PEMs.⁸ Of these 10 hits, 8 had relevant PEMs, which we searched for additional PEMs about bone and soft-tissue cancers and related conditions. We also conducted a Google search of the most common bone sarcoma and soft-tissue sarcoma, osteosarcoma and malignant fibrous histiocytoma, respectively, and found 2 additional websites with relevant PEMs. LaCoursiere and colleagues³⁵ surveyed cancer patients who used the Internet and found that they preferred WebMD (Webmd.com) and Medscape (Medscape.com) as sources for content about their medical condition.³⁵ WebMD had been identified in the Google search, and we gathered the PEMs from Medscape also. It is worth noting that some of these websites are written for patients as well as clinicians.

Text from these PEMs were copied and pasted into separate Microsoft Word documents (Microsoft, Redmond, Washington). Advertisements, pictures, picture text, hyperlinks, copyright notices, page navigation links, paragraphs with no text, and any text that was not related to the given condition were deleted from the document to format the text for the readability software. Then, each Microsoft Word document was uploaded into the software package Readability Studio Professional (RSP) Edition Version 2012.1 for Windows (Oleander Software, Vandalia, Ohio). The 10 distinct readability instruments that were used to gauge the readability of each document were the Flesch Reading Ease score (FRE), the New Fog Count, the New Automated Readability Index, the Coleman-Liau Index (CLI), the Fry readability graph, the New Dale-Chall formula (NDC), the Gunning Frequency of Gobbledygook (Gunning FOG), the Powers-Sumner-Kearl formula, the Simple Measure of Gobbledygook (SMOG), and the Raygor Estimate Graph.

The FRE’s formula takes the average number of words per sentence and average number of syllables per word to compute a score ranging from 0 to 100 with 0 being the hardest to read.³⁶ The New Fog Count tallies the number of sentences, easy words, and hard words (polysyllables) to calculate the grade level of the document.³⁷ The New Automated Readability Index takes the average characters per word and average words per sentence to calculate a grade level for the document.³⁷ The CLI randomly samples a few hundred words from the document, averages the number of letters and sentences per sample, and calculates an estimated grade level.³⁸ The Fry readability graph selects samples of 100 words from the document, averages the number of syllables and sentences per 100 words, plots these data points on a graph, with the intersection determining the reading level.³⁹ The NDC uses a list of 3000 familiar words that most fourth-grade students know.⁴⁰ The percentage of difficult words, which are not on the list of familiar words, and the average sentence length in words are used to calculate the reading grade level of the document. The Gunning FOG uses the average sentence length in words and the percentage of hard words from a sample of at least 100 words to determine the reading grade level of the document.⁴¹ The Powers-Sumner-Kearl formula uses the average sentence length and percentage of monosyllables from a 100-word sample passage to calculate the reading grade level.⁴² The SMOG formula counts the number of polysyllabic words from 30 sentences and calculates the reading grade level of the document.⁴³ In contrast to other formulas that test for 50% to 75% comprehension, the SMOG formula tests for 100% comprehension. As a result, the SMOG formula generally assigns a reading level 2 grades higher than the Dale-Chall level. The Raygor Estimate Graph selects a 100-word passage, counts the number of sentences and number of words with 6 or more letters, and plots the 2 variables on a graph to determine the reading grade level.⁴⁴ The software package calculated the results from each reading instrument and reported the mean grade level score
for each document.

Results

We identified a total of 72 websites with relevant PEMs and included them in this study. Of these 72 websites, 36 websites were academic training centers, 10 were Google search hits, and 21 were from the Sarcoma Alliance list of sarcoma specialists. The remaining 5 websites were AAOS, Bonetumor.org, Sarcoma Alliance, Sarcoma Foundation of America, and Medscape. A list of conditions and treatments that were considered relevant PEMs is found in Appendix 1. A total of 774 articles were obtained from the 72 websites.

None of the websites had a mean readability score of 7 (seventh grade) or lower (Figures 1A, 1B). Mid-America Sarcoma Institute’s PEMs had the lowest mean readability score, 8.9. The lowest readability score was 5.3, which the New Fog Count readability instrument calculated for Vanderbilt University Medical Center’s (VUMC’s) PEMs (Appendix 2). The mean readability score of all websites was 11.4 (range, 8.9-15.5) (Appendix 2).

Seventy of 72 websites (97%) had PEMs that were fairly difficult or difficult, according to the FRE analysis (Figure 2). The American Cancer Society and Mid-America Sarcoma Institute had PEMs that were written in plain English. Sixty-nine of 72 websites (96%) had PEMs with a readability score of 10 or higher, according to the Raygor readability estimate (Figure 3). Using this instrument, the scores of the American Cancer Society and the University of Pennsylvania–Joan Karnell Cancer Center were 9; Mid-America Sarcoma Institute’s score was 8. 

Discussion

Many cancer patients have turned to websites and online PEMs to gather health information about their condition.^10-17 Basch and colleagues¹⁰ reported almost a decade ago that 44% of cancer patients, as well as 60% of their companions, used the Internet to find cancer-related information.¹⁰ When LaCoursiere and colleagues³⁵ surveyed cancer patients, they found that patients handled their condition better and had less anxiety and uncertainty after using the Internet to find health information and support.³⁵ In addition, many orthopedic patients, specifically 46% of orthopedic community outpatients,⁴⁵ consult the Internet for information about their condition and future surgical procedures.^46,47

This study comprehensively evaluated the readability of online PEMs of bone and soft-tissue sarcomas and related conditions by using 10 different readability instruments. After identifying 72 websites and 774 articles, we found that all 72 websites’ PEMs had a mean readability score that did not meet the NIH recommendation of writing PEMs at a sixth- to seventh-grade reading level. These results are consistent with studies evaluating the readability of online PEMs related to other cancer conditions^21-25 and other orthopedic conditions.^26-31

The combination of low health literacy of many US adults and high reading grade levels of the majority of online PEMs is not conducive to patients’ better understanding their condition(s). Even individuals with high reading skills prefer information that is simpler to read.⁴⁸ In many areas of medicine, there is evidence that patients’ understanding of their condition has a positive impact on health outcomes, well-being, and the patient–physician relationship.^49-61 Regarding cancer patients, Davis and colleagues⁵⁴ and Peterson and colleagues⁵⁷ showed that lower health literacy contributes to less knowledge and lower rates of breast⁵⁴ and colorectal cancer⁵⁷ screening tests. Even low health literacy of family caregivers of cancer patients can result in increased stress and lack of communication of important medical information between caregiver and physician.⁵² Among cancer patients, poor health literacy has been associated with mental distress⁶⁰ as well as decreased compliance with treatment and lower involvement in clinical trials.⁵⁵

The disparity between patients’ health literacy and the readability of online PEMs needs to be addressed by finding methods to improve patients’ understanding of their condition and to lower the readability scores of online PEMs. Better communication between patient and physician may improve patients’ comprehension of their condition and different aspects of their care.^59,62-66 Doak and colleagues⁶³ recommend giving cancer patients the most important information first; presenting information to patients in smaller doses; intermittently asking patients questions; and incorporating graphs, tables, and drawings into communication with patients.⁶³ Additionally, allowing patients to repeat information they have just received/heard to the physician is another useful tool to improve patient education.^62,64-66

Another way to address the disparity between patients’ health literacy and the readability of online PEMs is to reduce the reading grade level of existing PEMs. According to results from this study and others, the majority of online PEMs are above the reading grade level of a significant number of US adults. Many available and inexpensive readability instruments allow authors to assess their articles’ readability. Many writing guidelines also exist to help authors improve the readability of their PEMs.^20,64,67-71 Living Word Vocabulary⁷⁰ and Plain Language⁷¹ help authors replace complex words or medical terms with simpler words.²⁹ Visual aids, audio, and video help patients with low health literacy remember the information.⁶⁴

Efforts to improve PEM readability are effective. Of all the websites reviewed, VUMC was identified as having PEMs with the lowest readability score (5.3). This score was reported by the New Fog Count readability instrument, which accounts for the number of sentences, easy words, and hard words. In 2011, VUMC formed the Department of Patient Education to review and update its online and printed PEMs to make sure patients could read them.⁷² Additionally, the mean readability scores of the websites of the National Cancer Institute and MedlinePlus are in the top 50% of the websites included in this study. The NIH sponsors both sites, which follow the NIH guidelines for writing online PEMs at a reading level suitable for individuals with lower health literacy.²⁰ These materials serve as potential models to improve the readability of PEMs, and, thus, help patients to better understand their condition, medical procedures, and/or treatment options.

To illustrate ways to improve the reading grade level of PEMs, we used the article “Ewing’s Sarcoma” from the AAOS website⁷³ and followed the NIH guidelines to improve the reading grade level of the article.²⁰ We identified complex words and defined them at an eighth-grade reading level. If that word was mentioned later in the article, simpler terminology was used instead of the initial complex word. For example, Ewing’s sarcoma was defined early and then referred to as bone tumor later in the article. We also identified every word that was 3 syllables or longer and used Microsoft Word’s thesaurus to replace those words with ones that were less than 3 syllables. Lastly, all sentences longer than 15 words were rewritten to be less than 15 words. After making these 3 changes to the article, the mean reading grade level dropped from 11.2 to 7.3. 

This study has limitations. First, some readability instruments evaluate the number of syllables per word or polysyllabic words as part of their formula and, thus, can underestimate or overestimate the reading grade level of a document. Some readability formulas consider medical terms such as ulna, femur, or carpal as “easy” words because they have 2 syllables, but many laypersons may not comprehend these words. On the other hand, some readability formulas consider medical terms such as medications, diagnosis, or radiation as “hard” words because they contain 3 or more syllables, but the majority of laypersons likely comprehend these words. Second, the reading level of the patient population accessing those online sites was not assessed. Third, the readability instruments in this study did not evaluate the accuracy of the content, pictures, or tables of the PEMs. However, using 10 readability instruments allowed evaluation of many different readability aspects of the text. Fourth, because some websites identified in this study, such as Bonetumor.org, were written for patients as well as clinicians, the reading grade level of these sites may be higher than that of those sites written just for patients.

Conclusion

Because many orthopedic cancer patients rely on the Internet as a source of information, the need for online PEMs to match the reading skills of the patient population who accesses them is vital. However, this study shows that many organizations, academic training centers, and other entities need to update their online PEMs because all PEMs in this study had a mean readability grade level higher than the NIH recommendation. Further research needs to evaluate the effectiveness of other media, such as video, illustrations, and audio, to provide health information to patients. With many guidelines available that provide plans and advice to improve the readability of PEMs, research also must assess the most effective plans and advice in order to allow authors to focus their attention on 1 set of guidelines to improve the readability of their PEMs.