Jonathan E. Sevransky, MD

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.