Alzheimer's assessment scale may not be accurate enough

Portions of one of the most commonly used tests to measure cognitive performance in Alzheimer’s disease trials may be too easy and may not accurately assess the range of patients’ cognitive abilities or detect their change over time, according to two complementary studies.

Analyses of Alzheimer’s Disease Assessment Scale – Cognitive Behavior Section (ADAS-Cog) scores measured in 193 patients with mild disease who participated in ADNI (the Alzheimer’s Disease Neuroimaging Initiative) over the course of 2 years detected limitations of the scale that could be improved, reported Dr. Jeremy Hobart of the Clinical Neurology Research Group at Plymouth (England) University Peninsula Schools of Medicine and Dentistry, and his colleagues.

The investigators used observational data from ADNI to show that out of 675 measurements made at time points of 0, 6, 12, and 24 months, data from ADAS-Cog total scores spanned the entire range of the scale and had no floor or ceiling effects that would reduce its ability to measure changes and differences in lower-functioning or higher-functioning patients, respectively. However, 8 of the scale’s 11 components (all except for word recall, word recognition, and orientation) had statistically significant ceiling effects with a skewed distribution of scores (Alzheimers Dement. 2012 [doi:10.1016/j.jalz.2012.08.005]). The mean age of patients was 74 years, 47% were female, and participants had a mean Mini-Mental State Examination score of 23 across all time points.

These results reproduce those that the investigators obtained in a previous psychometric evaluation of the ADAS-Cog in patients who participated in a randomized, controlled clinical trial (J. Neurol. Neurosurg. Psychiatry 2010;81:1363-8). They noted that the results mean that "these components may underestimate cognitive performance differences in those with mild to moderate AD-type dementia. This may lead to problems in detecting clinical change."

Because often greater than three-fourths of the participants with mild Alzheimer’s disease in the ADNI study scored either 0 or 1 on the majority of ADAS-Cog components, Dr. Hobart and his associates remarked that this would mean that few or no cognitive problems were detected. "However, as there is almost certainly greater variance in patient ability, this finding points to a limitation in the ADAS-Cog score function – namely that the ADAS-Cog, in its current form, is not subtle enough to record and monitor variance in the mildest stages of AD-type dementia."

In a second study that analyzed the extent to which the ADAS-Cog accurately measured cognitive performance in the same sample of patients, the investigators tried to avoid the limitations imposed by classical means of assessing reliability and validity of scales. They did this by using a method called Rasch Measurement Theory, which is a mathematical model designed to analyze the extent to which the rating scale data meet certain conditions necessary for the scale to record accurate measurements of cognitive performance. It provides diagnostic information that can help to revise a scale by exposing anomalies in the scale that can be improved and then retested, Dr. Hobart and his colleagues explained.

The range of cognitive performance measured by the 11 ADAS-Cog components suboptimally targeted the range of cognitive performance observed in patients in the sample. In six of the components, the integer-based scoring method used to assign cognitive performance did not reflect a continuum of performance as it was intended to, but instead indicated that some component scores were much more likely to occur than others, meaning that a higher score on one of these components did not confirm more cognitive impairment.

These gaps in some of the components’ abilities to measure cognitive performance means that their precision is limited, and that the raw scores of the ADAS-Cog and the linear relationship they are thought to reflect is actually an S-shaped relationship in which a 1-point change in ADAS-Cog measurement varies across the range of the scale, such that it is highest at the extremes of the its range and lowest at the center of its range, the investigators said (Alzheimers Dement. 2012 [doi:10.1016/j.jalz.2012.08.006]).

Both studies were supported in part by grants from an anonymous foundation, the U.K. National Institute for Health Research, and the U.S. National Institutes of Health.

Portions of one of the most commonly used tests to measure cognitive performance in Alzheimer’s disease trials may be too easy and may not accurately assess the range of patients’ cognitive abilities or detect their change over time, according to two complementary studies.

Analyses of Alzheimer’s Disease Assessment Scale – Cognitive Behavior Section (ADAS-Cog) scores measured in 193 patients with mild disease who participated in ADNI (the Alzheimer’s Disease Neuroimaging Initiative) over the course of 2 years detected limitations of the scale that could be improved, reported Dr. Jeremy Hobart of the Clinical Neurology Research Group at Plymouth (England) University Peninsula Schools of Medicine and Dentistry, and his colleagues.

The investigators used observational data from ADNI to show that out of 675 measurements made at time points of 0, 6, 12, and 24 months, data from ADAS-Cog total scores spanned the entire range of the scale and had no floor or ceiling effects that would reduce its ability to measure changes and differences in lower-functioning or higher-functioning patients, respectively. However, 8 of the scale’s 11 components (all except for word recall, word recognition, and orientation) had statistically significant ceiling effects with a skewed distribution of scores (Alzheimers Dement. 2012 [doi:10.1016/j.jalz.2012.08.005]). The mean age of patients was 74 years, 47% were female, and participants had a mean Mini-Mental State Examination score of 23 across all time points.

These results reproduce those that the investigators obtained in a previous psychometric evaluation of the ADAS-Cog in patients who participated in a randomized, controlled clinical trial (J. Neurol. Neurosurg. Psychiatry 2010;81:1363-8). They noted that the results mean that "these components may underestimate cognitive performance differences in those with mild to moderate AD-type dementia. This may lead to problems in detecting clinical change."

Because often greater than three-fourths of the participants with mild Alzheimer’s disease in the ADNI study scored either 0 or 1 on the majority of ADAS-Cog components, Dr. Hobart and his associates remarked that this would mean that few or no cognitive problems were detected. "However, as there is almost certainly greater variance in patient ability, this finding points to a limitation in the ADAS-Cog score function – namely that the ADAS-Cog, in its current form, is not subtle enough to record and monitor variance in the mildest stages of AD-type dementia."

In a second study that analyzed the extent to which the ADAS-Cog accurately measured cognitive performance in the same sample of patients, the investigators tried to avoid the limitations imposed by classical means of assessing reliability and validity of scales. They did this by using a method called Rasch Measurement Theory, which is a mathematical model designed to analyze the extent to which the rating scale data meet certain conditions necessary for the scale to record accurate measurements of cognitive performance. It provides diagnostic information that can help to revise a scale by exposing anomalies in the scale that can be improved and then retested, Dr. Hobart and his colleagues explained.

The range of cognitive performance measured by the 11 ADAS-Cog components suboptimally targeted the range of cognitive performance observed in patients in the sample. In six of the components, the integer-based scoring method used to assign cognitive performance did not reflect a continuum of performance as it was intended to, but instead indicated that some component scores were much more likely to occur than others, meaning that a higher score on one of these components did not confirm more cognitive impairment.

These gaps in some of the components’ abilities to measure cognitive performance means that their precision is limited, and that the raw scores of the ADAS-Cog and the linear relationship they are thought to reflect is actually an S-shaped relationship in which a 1-point change in ADAS-Cog measurement varies across the range of the scale, such that it is highest at the extremes of the its range and lowest at the center of its range, the investigators said (Alzheimers Dement. 2012 [doi:10.1016/j.jalz.2012.08.006]).

Both studies were supported in part by grants from an anonymous foundation, the U.K. National Institute for Health Research, and the U.S. National Institutes of Health.

Portions of one of the most commonly used tests to measure cognitive performance in Alzheimer’s disease trials may be too easy and may not accurately assess the range of patients’ cognitive abilities or detect their change over time, according to two complementary studies.

Analyses of Alzheimer’s Disease Assessment Scale – Cognitive Behavior Section (ADAS-Cog) scores measured in 193 patients with mild disease who participated in ADNI (the Alzheimer’s Disease Neuroimaging Initiative) over the course of 2 years detected limitations of the scale that could be improved, reported Dr. Jeremy Hobart of the Clinical Neurology Research Group at Plymouth (England) University Peninsula Schools of Medicine and Dentistry, and his colleagues.

The investigators used observational data from ADNI to show that out of 675 measurements made at time points of 0, 6, 12, and 24 months, data from ADAS-Cog total scores spanned the entire range of the scale and had no floor or ceiling effects that would reduce its ability to measure changes and differences in lower-functioning or higher-functioning patients, respectively. However, 8 of the scale’s 11 components (all except for word recall, word recognition, and orientation) had statistically significant ceiling effects with a skewed distribution of scores (Alzheimers Dement. 2012 [doi:10.1016/j.jalz.2012.08.005]). The mean age of patients was 74 years, 47% were female, and participants had a mean Mini-Mental State Examination score of 23 across all time points.

These results reproduce those that the investigators obtained in a previous psychometric evaluation of the ADAS-Cog in patients who participated in a randomized, controlled clinical trial (J. Neurol. Neurosurg. Psychiatry 2010;81:1363-8). They noted that the results mean that "these components may underestimate cognitive performance differences in those with mild to moderate AD-type dementia. This may lead to problems in detecting clinical change."

Because often greater than three-fourths of the participants with mild Alzheimer’s disease in the ADNI study scored either 0 or 1 on the majority of ADAS-Cog components, Dr. Hobart and his associates remarked that this would mean that few or no cognitive problems were detected. "However, as there is almost certainly greater variance in patient ability, this finding points to a limitation in the ADAS-Cog score function – namely that the ADAS-Cog, in its current form, is not subtle enough to record and monitor variance in the mildest stages of AD-type dementia."

In a second study that analyzed the extent to which the ADAS-Cog accurately measured cognitive performance in the same sample of patients, the investigators tried to avoid the limitations imposed by classical means of assessing reliability and validity of scales. They did this by using a method called Rasch Measurement Theory, which is a mathematical model designed to analyze the extent to which the rating scale data meet certain conditions necessary for the scale to record accurate measurements of cognitive performance. It provides diagnostic information that can help to revise a scale by exposing anomalies in the scale that can be improved and then retested, Dr. Hobart and his colleagues explained.

The range of cognitive performance measured by the 11 ADAS-Cog components suboptimally targeted the range of cognitive performance observed in patients in the sample. In six of the components, the integer-based scoring method used to assign cognitive performance did not reflect a continuum of performance as it was intended to, but instead indicated that some component scores were much more likely to occur than others, meaning that a higher score on one of these components did not confirm more cognitive impairment.

These gaps in some of the components’ abilities to measure cognitive performance means that their precision is limited, and that the raw scores of the ADAS-Cog and the linear relationship they are thought to reflect is actually an S-shaped relationship in which a 1-point change in ADAS-Cog measurement varies across the range of the scale, such that it is highest at the extremes of the its range and lowest at the center of its range, the investigators said (Alzheimers Dement. 2012 [doi:10.1016/j.jalz.2012.08.006]).

Both studies were supported in part by grants from an anonymous foundation, the U.K. National Institute for Health Research, and the U.S. National Institutes of Health.