Does exploratory factor analysis of the WJ3 test yield the test's claimed factor structure?

Dombrowski, S. C., & Watkins, M. W. (2013). Exploratory and higher order factor analysis of the WJ-III full test battery: A school-aged analysis.Psychological Assessment, 25(2), 442.‏   

This paper presents an exploratory factor analysis of the WJ3 test. 

Why is this interesting?  Because looking at new ways in which test are clustered into factors can open new ideas for test interpretation. 

What is exploratory factor analysis?

Factor analysis is a statistical method allowing researchers to reduce a large number of test measurements (that form an intricate picture that is difficult to interpret) to a small number of factors that are easier to interpret and explain.  Each factor is made of a cluster of some of the measures with which the researchers began the factor analysis.  When an exploratory factor analysis is performed, the researcher does not make any hypotheses about the factor structure.  When a confirmatory factor analysis is performed, the researcher makes a priori hypotheses about the factor structure, and these influence the results.

The authors of the WJ3 test published data on the test structure as a result of confirmatory factor analysis.  Dombrowski & Watkins re-analyzed the data of the original sample using exploratory factor analysis.  Were the results of the exploratory factor analysis in line with the results of the confirmatory factor analysis?  And if not – what's the significance of this?

The WJ3 test has two batteries:  a cognitive battery and an achievement battery.  The cognitive battery includes tests that measure the following abilities:  fluid ability, comprehension knowledge, visual processing, auditory processing, short term memory, long term storage and retrieval and processing speed.  The achievement battery includes tests that measure reading, writing and math, and a few tests that measure aspects of comprehension knowledge and auditory processing.

Dombrowski & Watkins included in the analysis the first 20 tests of the cognitive battery and the entire achievement battery.  These tests were intended to measure nine CHC abilities or nine factors:  fluid ability, comprehension knowledge, visual processing, auditory processing, short term memory, long term storage and retrieval, processing speed, reading and writing and quantitative ability.  We are used to consider reading and writing ability and quantitative ability as achievement domains, but in the CHC model they are also part of intelligence.

The exploratory factor analysis revealed that the data for the 9-13 age group was clustered into six factors, and the data for the 14-19 age group was clustered into five factors.  Where did three factors at ages 9-13 and four factors at ages 14-19 disappear to?  I'll introduce the factors for the 9-13 age group, with my interpretations about the test clusters.  I named the clusters, according to the common denominator  I found in the tests belonging to each cluster.  In most cases I used the CHC ability names.  You can give the factors different names, if you find names that reflect better the content measured in the tests that belong to each cluster.

 

The first factor, that can be named comprehension knowledge, includes the following tests:

WHICH ABILITY WAS THE TEST INTENDED TO MEASURE	BATTERY	DESCRIPTION	TEST NAME
Comprehension knowledge	Achievement tests	The child listens to a story and repeats it word by word	Story memory
Comprehension knowledge	Achievement tests	The child answers questions about science, social studies and humanities.	Academic knowledge
Comprehension knowledge	Achievement tests	The child names pictures of objects.	Picture vocabulary
Comprehension knowledge	Cognitive tests	The child names pictures of objects, finds synonyms and antonyms for given words, and solves verbal analogies.	Verbal Comprehension
Comprehension knowledge	Cognitive tests	The child answers questions about the locations of objects and the uses of objects.	General information
Long term storage and retrieval	Achievement tests	The child repeats the stories he heard and repeated in the "story memory" test.	Story memory delayed
Comprehension knowledge	Achievement tests	The child completes a missing word at the end of a sentence that is read to him	Oral comprehension
Reading and writing Comprehension knowledge	Achievement tests	The child finds synonyms and antonyms for given words, and solves verbal analogies.	Reading  vocabulary
Comprehension knowledge	Achievement tests	The child points at parts of a picture according to directions given to him	Understanding directions
Reading and writing	Achievement tests	The child points at pictures that fit a sentence, or reads a short passage and identifies a missing key word in it.	Passage comprehension
Long term storage and retrieval	Cognitive tests	The child retrieves as many words in a minute that belong to a specific category as he can.	Retrieval fluency

We can see the extent to which story memory is influenced by language and comprehension knowledge.  "Passage comprehension" obviously requires comprehension knowledge.  As for "retrieval fluency", it is surprising to find it here, since this task doesn't measure the depth or the breadth of the lexicon, as is done in tests assessing comprehension knowledge.  This is because the categories (for instance, animals, things you can eat) are usually easy, so that every child can retrieve many instances of each category.  In order to assess comprehension knowledge in such a test, one needs to use categories that reflect the depth of the child's comprehension knowledge (for instance, naming as many capital cities as the child can in one minute).

A second factor, which can be named reading and writing, included the following tests:

WHICH ABILITY WAS THE TEST INTENDED TO MEASURE	BATTERY	DESCRIPTION	TEST NAME
Reading and writing	Achievement tests	The child spells words that are read to him	spelling
Reading and writing Auditory processing	Achievement tests	The child reads non-words.	Word attack
Reading and writing	Achievement tests	The child identifies letters and words	Letter–Word Identification
Reading and writing Auditory processing	Achievement tests	The child writes letters and letter combinations that fit sounds and sound combinations	Spelling of sounds
Reading and writing	Achievement tests	The child finds and corrects spelling, punctuation and capitalization mistakes in short passages	Editing
Reading and writing	Achievement tests	The child writes sentences according to specific demands	Writing samples
Reading and writing Processing speed	Achievement tests	The child is given three words and a picture.  He has to quickly write a sentence containing these three words.	Writing fluency
Reading and writing	Achievement tests	The child reads sentences as fast as possible and decides if they are true or false.	Reading fluency
Auditory processing	Achievement tests	The child rhymes given words, alters sounds within words and reverses the order of sounds within words.	Sound awareness
Reading and writing	Achievement tests	The child points at pictures that fit a sentence, or reads a short passage and identifies a missing key word in it.	Passage comprehension
Reading and writing	Achievement tests		handwriting
Quantitative ability	Achievement tests	The child quickly solves as many problems in addition, subtraction and multiplication as he can	Math fluency

The most salient things here are the classification of sound awareness and math fluency tests to this factor/ability.  As for sound awareness, I think this is the result of the inability of the statistical analysis to distinguish between cause and effect.  A child who has poor sound awareness will also have poor reading and writing scores.   This is because phonological awareness is casually linked with reading.  Good phonological awareness enables one to learn to read.  As for math fluency, I can hypothesize that performance on this test requires reading the problems that are presented in a written form.  That may be the reason this test behaves in a similar way to tests measuring reading.

A third factor which can be named processing speed contained the following tests:

WHICH ABILITY WAS THE TEST INTENDED TO MEASURE	BATTERY	DESCRIPTION	TEST NAME
Processing speed Reading and writing	Achievement tests	The child is given three words and a picture.  He has to quickly write a sentence containing these three words.	Writing fluency
Reading and writing	Achievement tests	The child reads sentences as fast as possible and decides whether they are true or false.	Reading fluency
Processing speed	Cognitive tests	The child quickly scans a series of drawings and decides which two drawings are related to each other.	Decision speed
Processing speed	Cognitive tests	The child quickly identifies 2 identical geometric shaped out of a series of shapes or identifies 2 identical numbers out of a series of numbers	Visual matching
Processing speed	Cognitive tests	The child quickly identifies 2 items that appear in a specific sequence out of a series of distracting items.	Pair cancellation
Processing speed Long term storage and retrieval	Cognitive tests	The child quickly identifies and names drawings of common objects.	Rapid picture naming
Long term storage and retrieval	Cognitive tests	The child retrieves as many words belonging to a specific category as he can.	Retrieval fluency
Quantitative ability	achievement tests	The child quickly solves as many problems in addition, subtraction and multiplication as he can.	Math fluency

Here I find two interesting things:  A.  The classification of the retrieval fluency test to processing speed and not to long term storage and retrieval.  This means that at least according to this analysis, the speed factor (and the comprehension knowledge factor as we saw above) are at least as dominant in this test as the retrieval factor.  B.  A mix of cause and effect:  writing fluency, reading fluency and math fluency are made possible (at least partly) as a result of intact processing speed.

A forth factor, that can be named fluid ability, contained the following tests:

WHICH ABILITY WAS THE TEST INTENDED TO MEASURE	BATTERY	DESCRIPTION	TEST NAME
Quantitative ability Fluid ability	Achievement tests	The child analyses and solves practical math problems by deciding which mathematical operation to use.	Applied problems
Fluid ability	Cognitive tests	The child applies given rules to solve problems.	Analysis and synthesis
Quantitative ability	Achievement tests	The test measures the child's familiarity with concepts, symbols and math vocabulary.  The child also completes mathematical series.	Quantitative concepts
Fluid ability	Cognitive tests	The child derives a rule according to which items are classified.	Concept formation
Quantitative ability	Achievement tests	The child solves problems in multiplication, division, addition and subtraction and higher math.	Calculation
Visual processing	Cognitive tests	The child chooses the set of parts that form a specific shape, by performing mental rotations.	Spatial relations
Short term memory	Cognitive tests	The child listens to series of numbers and repeats them backwards.	Numbers Reversed
Fluid ability Visual processing	Cognitive tests	The child traces a complex drawing without lifting the pencil or retracing parts that he had already traced.	Planning

Here we can see, in my view, the extent to which mathematical reasoning tests require fluid ability.  In order to transfer a word problem into math symbols, abstract thinking is needed.  In order to solve math series one needs to use induction (to understand the rule by which the series is composed and apply it to solve the series) and so on.  Another interesting thing we see here is the extent to which "Numbers Reversed" is a complex test that is highly loaded on fluid ability.  In this task the child has to create a strategy, divide attention between two sets (the numbers forward and backwards) and shift between them, and maintain a high level of effort for an extended period of time.  Lohman and  Lakin (reference below) note that there is a large overlap between fluid ability and working memory.  We also see in this cluster two tests that have a visual component:  "Spatial Relations" and "Planning".  It may be that the classification of "Spatial Relations" here also reflects the complexity of the task that includes mental rotations.

A fifth factor that can be named serial processing contained the following tests:

WHICH ABILITY WAS THE TEST INTENDED TO MEASURE	BATTERY	DESCRIPTION	TEST NAME
Comprehension knowledge	Achievement tests	The child points at objects in a picture according to instructions.	Understanding directions
Auditory processing	Achievement tests	The child rhymes given words, alters sounds within words and reverses the order of sounds within words.	Sound awareness
Auditory processing	Cognitive tests	The child listens to word divided into syllables, subsyllables and phonemes.  He has to recognize the words.	Sound Blending
Short term memory	Cognitive tests	The child repeats series of words in the same order he heard them.	Memory for words
Auditory processing.	Cognitive tests.	The child identifies words in which one or more phonemes are missing.	Incomplete words

Here it seems like the common denominator to all tests is the need to process sequences:  in the "Understanding Directions" test, the child performs a number of directions in the sequence they were given to him.  The  auditory processing tests require processing or manipulation of the sequence of sounds in words.  The "Memory for Words" test requires the repetition of a series of words in the order of their presentation. 

A sixth factor that can be named long term storage and retrieval or associative learning contains the following tests:

WHICH ABILITY WAS THE TEST INTENDED TO MEASURE	BATTERY	DESCRIPTION	TEST NAME
Long term storage and retrieval	Cognitive tests	The child learns pairs of words and symbols and "reads" sentences composed of the symbols.	Visual auditory learning
Long term storage and retrieval	Cognitive tests	At least half an hour after the initial learning of the pairs, the child "reads" again novel sentences composed of the symbols.	Visual auditory learning - delayed

In these tests, the child learns and retrieves pairs of symbols and words.  The rest of the tests that measure long term storage and retrieval were scattered to other factors:  "Story Memory" is classified to Comprehension Knowledge, "Rapid Picture Naming" is classified to Processing Speed and "Retrieval fluency" is classified both to Comprehension Knowledge and to Processing Speed. 

Which factors/abilities are missing and why?

 Visual processing, Short Term Memory, Auditory Processing, and Quantitative Ability are missing.

Two of the visual processing tests are found in the factor which we named "fluid".  The "Picture Recognition" test, where the child recognizes drawings of objects he saw previously in a series of drawings, was not classified to any factor.

Out of the short term memory tests, "Numbers Reversed" is in the Fluid factor.  "Memory for Words" was classified in the Serial Processing factor.  "Auditory Working Memory" (in which the child listens to series of words and numbers and repeats the words at the order of their presentation and then the numbers at the order of their presentation) was not classified to any factor. 

Another unclassified test was "Auditory Attention", in which the child has to point to drawings that represent given words while a background noise gets louder and louder.  This test was designed to measure Auditory Processing.  Other auditory processing tests were classified to the series processing factor.

Out of the quantitative ability tests, "Math Fluency" was moved to Processing Speed, "Applied Problems" and "Calculations" were moved to the Fluid factor.

The place of the full scale IQ score in the analysis:

In this exploratory factor analysis, the total and common variance accounted for by the higher order (g) factor dwarfed that apportioned to the lower order factors. At the individual subtest level, the variance accounted for by the higher order (g) factor exceeds that apportioned to any lower order factor.  This means that the full scale IQ is the most important factor that can be derived from the WJ3 test (and this is not surprising, this is the way it's supposed to be).  The contribution of the broad abilities to data explanations is marginal (this is surprising). 

The analysis presented here allows us to think a little differently about the ways in which we conceptualize the cognitive factors/abilities and the tests that belong to each factor.

Lohman, D. F., Lakin, J. M., Sternberg, R. J., & Kaufman, S. B. (2009).Reasoning and intelligence. Handbook of intelligence, 419-441.‏  https://faculty.education.uiowa.edu/docs/dlohman/Reasoning-and-Intell_Lohman-Lakin-102709.pdf