User login
During the influenza portion of the Feb. 21, 2018, Centers for Diseases Control and Prevention’s Advisory Committee on Immunization Practices meeting, two pleas from the audience asked the CDC/ACIP to make messages very clear about how protective influenza vaccine really is.
We hear apparently conflicting percentages from Australia, Canada, Europe, and the United States from the many stories/press releases in the news media and from official public health outlets. And the gloomiest ones get the most exposure.1 It can be confusing even for medical care providers who are supposed to advise families on such matters.
A key misunderstanding in many medical and lay news stories is about what vaccine effectiveness and vaccine efficacy really mean. What? Aren’t those the same thing? Nope. They are quite different. And are we sure of what those 95% confidence intervals (CI) mean? Let’s review the “math” so we can explain this to families.
Vaccine effectiveness (VE)2,3
The first thing to know is that the CDC and similar public health agencies in other countries do not report vaccine efficacy. Instead, the percentage reported is VE during (interim estimated VE) and just after (final adjusted VE) each influenza season. This means that VE is generally a retrospective analysis of data, most of which were collected prospectively. Further, VE is likely the worst case scenario. VE is a measure of real-world benefit to patients for whom vaccine is recommended, by analyzing specific geographically diverse populations (population-based) without excluding most underlying illness or comorbidities (note that immunosuppressed persons are excluded). Subjects in VE studies receive their vaccine in the real world and, therefore, vaccinees may receive their vaccines from any number of the usual outlets (e.g., primary care provider, urgent care or emergency department, public health department, pharmacy, school, or nursing home). There are multiple lots of multiple brands from multiple vaccine manufacturers. Children who need two doses of influenza vaccine do not necessarily receive those doses according to the package insert’s schedule. VE studies do not have the capability to confirm that vaccine was stored, handled, and administered in a precisely correct manner according to manufacturer’s and CDC’s recommendations.
VE is calculated using a “test-negative” (case-control) analysis of patients presenting with acute respiratory infections (ARIs). People who are not in vaccine research can find this methodology confusing. Briefly, the VE compares the odds of vaccination in ARIs due to confirmed influenza to the odds of vaccination in ARIs not due to influenza. Additional statistical tools can adjust VE for specific factors. VE is also calculated by factors of interest, such as age, gender, pregnancy, influenza type, region of the country, presence of asthma or other comorbidity, etc. Whether the VE value is the “truth in the universe” is related to having enough subjects in each analyzed group and the degree to which the studied populations actually represent the whole country. So, VE is more accurate when there are large subject numbers.
Remember also that VE is usually calculated from outpatients, so it does not really measure all the benefits of vaccination. Prevention rates for severe influenza (such as influenza hospitalizations) are higher but usually unavailable until after the entire season.
VE studies generally measure real-world and likely worst-case-scenario benefit for the overall population being protected against outpatient influenza medical visits.
Vaccine efficacy2,3
Vaccine efficacy measures how the vaccine performs under ideal circumstances in a regimented protocol in relatively normal hosts – likely the best-case-scenario benefit. Vaccine efficacy is the percent difference in confirmed influenza episodes in vaccinees getting the “experimental” vaccine vs. episodes in nonvaccinees (or vaccinees getting an established vaccine). Vaccine efficacy, therefore, is usually calculated based on prospective well-controlled studies under ideal circumstances in subjects who received their vaccines on time per the recommended schedule. Most such studies are performed on otherwise healthy children or adults, with most comorbidities excluded. The “experimental” vaccine is generally from a single manufacturer from a single lot, and chain-of-custody is well controlled. The vaccine is administered at selected research sites according to a strict protocol; vaccine storage is ensured to be as recommended.
Confidence intervals
To assess whether the “protection” is “significant,” the calculations derive 95% confidence intervals (CI). If the 95% CI range is wide, such as many tens of percents, then there is less confidence that the calculation is correct. And if the lower CI is less than 0, then the result is not significant. For example, a VE of 20% is not highly protective, but can be significant if the 95% CI ranges from 10 to 28 (the lower value of 10 is above zero). It would not be significant if the 95% CI lower limit was –10. Values for seasons 2004-2005 and 2005-2006 were similar to this. Consider however that a VE of 55% seems great, but may not be significant if the 95% CI range is –20 to 89 (the lower value is less than zero). In the ideal world, the VE would be greater than 50% and the 95% CI range would be tight with the lower CI value far above zero; for example, VE of 70% with 95% CI ranging from 60 to 80. The 2010-2011 season was close to this.
Type and age-specific VE
Aside from overall VE, there are subset analyses that can be revealing. This year there are the concerning mid-season VE estimates of approximately 25% for the United States and 17% in Canada, for one specific type, H3N2, which unfortunately has been the dominant circulating U.S. type. That number is what everybody seems to have focused on. But remember influenza B becomes dominant late in most seasons (increasing at the time of writing this article). Interim 2017-2018 VE for influenza B was in the mid 60% range, making the box plot near 40% overall.
Age-related VE analysis can show difference; for example, the best benefit for H3N2 this season has been in young children and the worst in elderly and 9- to 17-year-olds.
Take-home message
The simplest way to think of overall VE is that it is the real-world, worst-case-scenario value for influenza protection by vaccine against the several circulating types of influenza. 
Dr. Harrison is professor of pediatrics and pediatric infectious diseases at Children’s Mercy Hospitals and Clinics, Kansas City, Mo. Children’s Mercy Hospital receives grant funding for Dr. Harrison’s work as an investigator from GSK for MMR and rotavirus vaccine studies, from Merck for in vitro and clinical antibiotic studies, from Allergan for clinical antibiotic studies, from Pfizer for pneumococcal seroepidemiology studies, and from Regeneron for RSV studies. Dr. Harrison received support for travel and to present seroepidemiology data at one meeting. Email him at [email protected].
References
1. MMWR Weekly. 2017 Feb 17;66(6):167-71.
2. Dev Biol Stand. 1998;95:195-201.
3. Lancet Infect Dis. 2012 Jan;12(1):36-44.
During the influenza portion of the Feb. 21, 2018, Centers for Diseases Control and Prevention’s Advisory Committee on Immunization Practices meeting, two pleas from the audience asked the CDC/ACIP to make messages very clear about how protective influenza vaccine really is.
We hear apparently conflicting percentages from Australia, Canada, Europe, and the United States from the many stories/press releases in the news media and from official public health outlets. And the gloomiest ones get the most exposure.1 It can be confusing even for medical care providers who are supposed to advise families on such matters.
A key misunderstanding in many medical and lay news stories is about what vaccine effectiveness and vaccine efficacy really mean. What? Aren’t those the same thing? Nope. They are quite different. And are we sure of what those 95% confidence intervals (CI) mean? Let’s review the “math” so we can explain this to families.
Vaccine effectiveness (VE)2,3
The first thing to know is that the CDC and similar public health agencies in other countries do not report vaccine efficacy. Instead, the percentage reported is VE during (interim estimated VE) and just after (final adjusted VE) each influenza season. This means that VE is generally a retrospective analysis of data, most of which were collected prospectively. Further, VE is likely the worst case scenario. VE is a measure of real-world benefit to patients for whom vaccine is recommended, by analyzing specific geographically diverse populations (population-based) without excluding most underlying illness or comorbidities (note that immunosuppressed persons are excluded). Subjects in VE studies receive their vaccine in the real world and, therefore, vaccinees may receive their vaccines from any number of the usual outlets (e.g., primary care provider, urgent care or emergency department, public health department, pharmacy, school, or nursing home). There are multiple lots of multiple brands from multiple vaccine manufacturers. Children who need two doses of influenza vaccine do not necessarily receive those doses according to the package insert’s schedule. VE studies do not have the capability to confirm that vaccine was stored, handled, and administered in a precisely correct manner according to manufacturer’s and CDC’s recommendations.
VE is calculated using a “test-negative” (case-control) analysis of patients presenting with acute respiratory infections (ARIs). People who are not in vaccine research can find this methodology confusing. Briefly, the VE compares the odds of vaccination in ARIs due to confirmed influenza to the odds of vaccination in ARIs not due to influenza. Additional statistical tools can adjust VE for specific factors. VE is also calculated by factors of interest, such as age, gender, pregnancy, influenza type, region of the country, presence of asthma or other comorbidity, etc. Whether the VE value is the “truth in the universe” is related to having enough subjects in each analyzed group and the degree to which the studied populations actually represent the whole country. So, VE is more accurate when there are large subject numbers.
Remember also that VE is usually calculated from outpatients, so it does not really measure all the benefits of vaccination. Prevention rates for severe influenza (such as influenza hospitalizations) are higher but usually unavailable until after the entire season.
VE studies generally measure real-world and likely worst-case-scenario benefit for the overall population being protected against outpatient influenza medical visits.
Vaccine efficacy2,3
Vaccine efficacy measures how the vaccine performs under ideal circumstances in a regimented protocol in relatively normal hosts – likely the best-case-scenario benefit. Vaccine efficacy is the percent difference in confirmed influenza episodes in vaccinees getting the “experimental” vaccine vs. episodes in nonvaccinees (or vaccinees getting an established vaccine). Vaccine efficacy, therefore, is usually calculated based on prospective well-controlled studies under ideal circumstances in subjects who received their vaccines on time per the recommended schedule. Most such studies are performed on otherwise healthy children or adults, with most comorbidities excluded. The “experimental” vaccine is generally from a single manufacturer from a single lot, and chain-of-custody is well controlled. The vaccine is administered at selected research sites according to a strict protocol; vaccine storage is ensured to be as recommended.
Confidence intervals
To assess whether the “protection” is “significant,” the calculations derive 95% confidence intervals (CI). If the 95% CI range is wide, such as many tens of percents, then there is less confidence that the calculation is correct. And if the lower CI is less than 0, then the result is not significant. For example, a VE of 20% is not highly protective, but can be significant if the 95% CI ranges from 10 to 28 (the lower value of 10 is above zero). It would not be significant if the 95% CI lower limit was –10. Values for seasons 2004-2005 and 2005-2006 were similar to this. Consider however that a VE of 55% seems great, but may not be significant if the 95% CI range is –20 to 89 (the lower value is less than zero). In the ideal world, the VE would be greater than 50% and the 95% CI range would be tight with the lower CI value far above zero; for example, VE of 70% with 95% CI ranging from 60 to 80. The 2010-2011 season was close to this.
Type and age-specific VE
Aside from overall VE, there are subset analyses that can be revealing. This year there are the concerning mid-season VE estimates of approximately 25% for the United States and 17% in Canada, for one specific type, H3N2, which unfortunately has been the dominant circulating U.S. type. That number is what everybody seems to have focused on. But remember influenza B becomes dominant late in most seasons (increasing at the time of writing this article). Interim 2017-2018 VE for influenza B was in the mid 60% range, making the box plot near 40% overall.
Age-related VE analysis can show difference; for example, the best benefit for H3N2 this season has been in young children and the worst in elderly and 9- to 17-year-olds.
Take-home message
The simplest way to think of overall VE is that it is the real-world, worst-case-scenario value for influenza protection by vaccine against the several circulating types of influenza.
Dr. Harrison is professor of pediatrics and pediatric infectious diseases at Children’s Mercy Hospitals and Clinics, Kansas City, Mo. Children’s Mercy Hospital receives grant funding for Dr. Harrison’s work as an investigator from GSK for MMR and rotavirus vaccine studies, from Merck for in vitro and clinical antibiotic studies, from Allergan for clinical antibiotic studies, from Pfizer for pneumococcal seroepidemiology studies, and from Regeneron for RSV studies. Dr. Harrison received support for travel and to present seroepidemiology data at one meeting. Email him at [email protected].
References
1. MMWR Weekly. 2017 Feb 17;66(6):167-71.
2. Dev Biol Stand. 1998;95:195-201.
3. Lancet Infect Dis. 2012 Jan;12(1):36-44.
During the influenza portion of the Feb. 21, 2018, Centers for Diseases Control and Prevention’s Advisory Committee on Immunization Practices meeting, two pleas from the audience asked the CDC/ACIP to make messages very clear about how protective influenza vaccine really is.
We hear apparently conflicting percentages from Australia, Canada, Europe, and the United States from the many stories/press releases in the news media and from official public health outlets. And the gloomiest ones get the most exposure.1 It can be confusing even for medical care providers who are supposed to advise families on such matters.
A key misunderstanding in many medical and lay news stories is about what vaccine effectiveness and vaccine efficacy really mean. What? Aren’t those the same thing? Nope. They are quite different. And are we sure of what those 95% confidence intervals (CI) mean? Let’s review the “math” so we can explain this to families.
Vaccine effectiveness (VE)2,3
The first thing to know is that the CDC and similar public health agencies in other countries do not report vaccine efficacy. Instead, the percentage reported is VE during (interim estimated VE) and just after (final adjusted VE) each influenza season. This means that VE is generally a retrospective analysis of data, most of which were collected prospectively. Further, VE is likely the worst case scenario. VE is a measure of real-world benefit to patients for whom vaccine is recommended, by analyzing specific geographically diverse populations (population-based) without excluding most underlying illness or comorbidities (note that immunosuppressed persons are excluded). Subjects in VE studies receive their vaccine in the real world and, therefore, vaccinees may receive their vaccines from any number of the usual outlets (e.g., primary care provider, urgent care or emergency department, public health department, pharmacy, school, or nursing home). There are multiple lots of multiple brands from multiple vaccine manufacturers. Children who need two doses of influenza vaccine do not necessarily receive those doses according to the package insert’s schedule. VE studies do not have the capability to confirm that vaccine was stored, handled, and administered in a precisely correct manner according to manufacturer’s and CDC’s recommendations.
VE is calculated using a “test-negative” (case-control) analysis of patients presenting with acute respiratory infections (ARIs). People who are not in vaccine research can find this methodology confusing. Briefly, the VE compares the odds of vaccination in ARIs due to confirmed influenza to the odds of vaccination in ARIs not due to influenza. Additional statistical tools can adjust VE for specific factors. VE is also calculated by factors of interest, such as age, gender, pregnancy, influenza type, region of the country, presence of asthma or other comorbidity, etc. Whether the VE value is the “truth in the universe” is related to having enough subjects in each analyzed group and the degree to which the studied populations actually represent the whole country. So, VE is more accurate when there are large subject numbers.
Remember also that VE is usually calculated from outpatients, so it does not really measure all the benefits of vaccination. Prevention rates for severe influenza (such as influenza hospitalizations) are higher but usually unavailable until after the entire season.
VE studies generally measure real-world and likely worst-case-scenario benefit for the overall population being protected against outpatient influenza medical visits.
Vaccine efficacy2,3
Vaccine efficacy measures how the vaccine performs under ideal circumstances in a regimented protocol in relatively normal hosts – likely the best-case-scenario benefit. Vaccine efficacy is the percent difference in confirmed influenza episodes in vaccinees getting the “experimental” vaccine vs. episodes in nonvaccinees (or vaccinees getting an established vaccine). Vaccine efficacy, therefore, is usually calculated based on prospective well-controlled studies under ideal circumstances in subjects who received their vaccines on time per the recommended schedule. Most such studies are performed on otherwise healthy children or adults, with most comorbidities excluded. The “experimental” vaccine is generally from a single manufacturer from a single lot, and chain-of-custody is well controlled. The vaccine is administered at selected research sites according to a strict protocol; vaccine storage is ensured to be as recommended.
Confidence intervals
To assess whether the “protection” is “significant,” the calculations derive 95% confidence intervals (CI). If the 95% CI range is wide, such as many tens of percents, then there is less confidence that the calculation is correct. And if the lower CI is less than 0, then the result is not significant. For example, a VE of 20% is not highly protective, but can be significant if the 95% CI ranges from 10 to 28 (the lower value of 10 is above zero). It would not be significant if the 95% CI lower limit was –10. Values for seasons 2004-2005 and 2005-2006 were similar to this. Consider however that a VE of 55% seems great, but may not be significant if the 95% CI range is –20 to 89 (the lower value is less than zero). In the ideal world, the VE would be greater than 50% and the 95% CI range would be tight with the lower CI value far above zero; for example, VE of 70% with 95% CI ranging from 60 to 80. The 2010-2011 season was close to this.
Type and age-specific VE
Aside from overall VE, there are subset analyses that can be revealing. This year there are the concerning mid-season VE estimates of approximately 25% for the United States and 17% in Canada, for one specific type, H3N2, which unfortunately has been the dominant circulating U.S. type. That number is what everybody seems to have focused on. But remember influenza B becomes dominant late in most seasons (increasing at the time of writing this article). Interim 2017-2018 VE for influenza B was in the mid 60% range, making the box plot near 40% overall.
Age-related VE analysis can show difference; for example, the best benefit for H3N2 this season has been in young children and the worst in elderly and 9- to 17-year-olds.
Take-home message
The simplest way to think of overall VE is that it is the real-world, worst-case-scenario value for influenza protection by vaccine against the several circulating types of influenza.
Dr. Harrison is professor of pediatrics and pediatric infectious diseases at Children’s Mercy Hospitals and Clinics, Kansas City, Mo. Children’s Mercy Hospital receives grant funding for Dr. Harrison’s work as an investigator from GSK for MMR and rotavirus vaccine studies, from Merck for in vitro and clinical antibiotic studies, from Allergan for clinical antibiotic studies, from Pfizer for pneumococcal seroepidemiology studies, and from Regeneron for RSV studies. Dr. Harrison received support for travel and to present seroepidemiology data at one meeting. Email him at [email protected].
References
1. MMWR Weekly. 2017 Feb 17;66(6):167-71.
2. Dev Biol Stand. 1998;95:195-201.
3. Lancet Infect Dis. 2012 Jan;12(1):36-44.