McGill, R. J., & Busse, R. T.
(2015). Incremental validity of the WJ III COG: Limited predictive effects
beyond the GIA-E. School
Psychology Quarterly, 30(3),
353. https://pdfs.semanticscholar.org/f5b5/d70077a1b7747a31bbcd5fb7b7dfcc38c2a3.pdf
What predicts achievement
better: g or broad abilities?
Well, it
depends on the model of intelligence that you're using, and on the way the
statistical analysis was done in the research on which you base your
conclusion.
The WJ III COG examiner manual encourages primary interpretation at
the broad ability level (e.g., CHC-related cluster scores). Because linking performance
in reading/writing/math to the state of the child's cognitive
abilities is a major use of intelligence tests, examining relationships between
WJ III COG cluster scores and external achievement measures is important. These
examinations are also critically important for evaluating the tenability of
several models that have been proposed for use in the identification of
specific learning disabilities (SLD) in children and adolescents. These and
similar models utilize lower-order scores, such as the WJ III COG broad ability
clusters, as a critical component for determining whether or not an individual
has a learning disability.
According to Flanagan's model, a child is learning disabled if:
A. he has significantly poor
reading/writing/math achievement.
B. he has one (or two)
significantly low broad ability scores.
C. the low broad ability scores
can explain the child's poor
performance in reading/writing/math.
D. the child's other broad
abilities are average or above average.
E. excluding factors (like
insufficient or inappropriate instruction or emotional problems) are not better
explanations of the poor reading/writing/math achievement.
Shortly after the publication of the WJ III COG, McGrew and his
colleagues utilized multiple regression to examine predictive relationships
between WJ III COG CHC clusters and standardized reading, writing and math
measures. Their analyses provided evidence for differential predictive effects
across the age span for specific CHC clusters.
I'd reviewed one of McGrew's studies in a prior post. Here are slides from that post (click to enlarge).
These slides contain valuable
information that can help us plan the diagnostic process, identify the child's difficulties
and plan an intervention.
But McGill and Busse argue that these studies did not control for
potential effects of the common variance shared by mental measures. The common variance is a manifestation of
g. Each subtest and broad ability
measures both g and the specific thing it is supposed to measure. If a broad ability is "saturated"
with g, it is a good measure of g but a poorer measure of the unique construct
it's supposed to measure.
To investigate the tenability of the recommendation for
practitioners to interpret primarily at the broad ability level, it is
necessary to examine the incremental predictive validity provided by the broad
abilities after controlling for the effects of variance already accounted for
by the full scale score. That is, the extent to which broad abilities add
meaningful information beyond what we get from the full scale IQ score.
The participants in this study were children and adolescents ages
6–0 to 18–11 (n = 4,722) drawn from the standardization sample for the WJ III
COG and the WJ III ACH. The WJ III COG measures
7 broad abilities: Comprehension Knowledge, Fluid Reasoning, Long Term Storage
and Retrieval, Short Term Memory, Visual Processing, Auditory Processing and Processing
Speed. The General Intellectual Ability – Extended (GIA-E) score is composed of
14 subtests, 2 subtests for each broad ability.
McGill and Busse used Hierarchical multiple regression analysis to analyze
the data. In this procedure, the full
scale score is entered first into a regression equation followed by the lower
order factor or cluster scores to predict a criterion achievement variable.
This entry technique allows for the predictive effects of the cluster scores to
be assessed while controlling for the effects of the full scale score.
The authors found that GIA-E accounted for statistically significant
portions of each of the WJ III ACH cluster scores: Broad Reading, Basic
Reading, Reading Comprehension, Broad Mathematics, Math Calculation, Math
Reasoning, Broad Written Language, Basic Writing, Written Expression, Oral
Expression, and Listening Comprehension (I'm not sure oral expression and
listening comprehension are really achievement areas. I think they belong more to the comprehension
knowledge cluster and they are also affected by fluid reasoning). The GIA-E accounted for 29% (Math Calculation Skills) to 56%
(Listening Comprehension; Mdn 46%) of the criterion variable variance.
CHC clusters entered jointly
into the second block of the regression equations accounted for 2% (Math
Reasoning) to 23% (Oral Expression; Mdn 5%) of the incremental variance. The 23% incremental variance in Oral
Expression was predicted by the Comprehension Knowledge cluster. But even this unique outcome is
dubious: oral expression tests measure
linguistic competency and vocabulary, that is – Comprehension Knowledge. So it's hardly surprising that Comprehension Knowledge
predicts Comprehension Knowledge…None of the other broad abilities accounted for more than 5% of
achievement variance beyond GIA-E.
Although the CHC clusters contributed significant portions of
incremental achievement variance beyond the effects of the GIA-E, effect size
estimates were negligible. The results
from the current study indicate that practitioners who interpret CHC cluster
scores on the WJ III COG, without accounting for the effects of the GIA-E risk
overestimating the predictive effects of various CHC-related abilities
These results are fairly consistent with those that have been
obtained from other cognitive measures like the Wechsler test.
But how does all this fit with
McGrew's results presented in the slides above?
McGill and Busse write that
reverse entry of the independent variables, in this case entering the broad
ability clusters first, would result in the clusters accounting for
approximately the same variance proportions that were attributed to the GIA-E
in this study. Consequently,
the GIA-E would provide little incremental prediction. Order of entry
arbitrarily determines whether scores such as the GIA mean everything or
nothing. However,
order of entry is not an arbitrary process and must be determined a priori
according to expected theoretical relationships between the variables and
causal priority. Contemporary intelligence theory (e.g., CHC) and the WJ III
COG structural model support entering the GIA-E before the clusters because the
cluster scores are both theoretically and statistically subordinate to the
GIA-E. Reverse entry conflicts with existing intelligence theory and violates
the scientific law of parsimony (if you can predict something with one variable
with the same level of success as with many variables, prefer the one over the
many).
The CHC model is an integration between the Cattell and Horn model
and Carroll's model. Cattell and Horn
presented a model of intelligence that included broad abilities but did not
include g. Carroll presented a model
that included both g and broad abilities. To this day, many questions remain as
to whether g reflects an actual latent ability or is merely a statistical
artifact resulting from the tendency for all tests of mental ability to be
positively correlated.
So, if I believe there's no g
(like Cattell and Horn), the first thing I'll enter into the regression
equation would be the broad abilities, and the results I'd get would be that
the broad abilities predict achievement pretty well. On the other hand, If I believe in g (like
Carroll), I'll enter g first and find that it's the best predictor of
achievement and that broad abilities do not add any meaningful incremental
value…
What's more, it's obvious that
FSIQ predicts achievement. Obviously, it
predicts achievement better than any broad ability, since it embodies the
influence of all broad abilities. But
this is not the interesting question.
What we want to know is to what extent do the components of g,
that is the broad abilities, predict achievement. I think this is a good theoretical reason to
enter broad abilities first into the regression equation whether we believe in
g or not. We don't want to use g to
predict achievement because it's too broad and doesn't lead to meaningful
interventions.
McGill and Busse point out
that incremental validity researchers have largely relied on archived
standardization data to assess the predictive effects of cognitive test scores.
This is problematic given that the two incremental validity studies that have
been conducted using data obtained from clinical samples have found
significantly diminished effects associated with the general factor with
greater portions of achievement variance accounted for by factor-level scores.
Additionally whereas the cognitive variables consistently accounted
for large portions of achievement variance, approximately half of the variance
in the WJ III ACH variables was left unpredicted in this study. What explains this additional
variance? Maybe noncognitive variables
like motivation or effort.
