Cognition or personality?

Raymond Cattell (the first C in CHC…) considered intelligence to be a facet of personality.  Intelligence/reasoning is one of the 16 personality factors in Cattell's model. 

We know that responses to items in tests that measure intelligence and cognitive abilities also reflect facets of personality, and that responses to items in tests that assess personality also reflect cognitive abilities.

The American Psychological Association (APA) defines personality as “individual differences in characteristic patterns of thinking, feeling, and behaving”.  APA’s 1996 Intelligence Task Force likewise provides a definition of intelligence as “individual differences in the ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought”.

Intelligence, personality and the assessment method

For a combination of historical, accidental, and practical reasons, two broad approaches to measuring these two constructs have emerged and come to dominate how we think of them. For personality, the dominant methodology has to do with endorsements of descriptions of characteristic behavior, thoughts, beliefs, or attitudes. The endorsements can be done by the self or by others—peers, teachers, or supervisors. The essence of the method is that it involves evaluating the target’s “characteristic patterns of thinking, feeling, and behaving” represented by descriptions.  For intelligence, the dominant method is the standardized test, with a problem and response format (multiple choice, short answer, and essay), scored as right or wrong, or in some cases, partially right.

However, intelligence can just as easily as personality be evaluated with the statement endorsement methodology. That is, rather than giving a test, we can ask examinees their level of agreement to statements such as “I understand complex ideas”, “I adapt effectively to the environment”, “I learn from experience”, or “I engage in various forms of reasoning to overcome obstacles by taking thought".  The correlation between self-estimates of intelligence and intelligence test scores is about r = 0.33.

In order to have meaning, the assessment of personality via questionnaires should be useful.  It should explain, for example, interpersonal differences in the ability to learn new things and solve new problems, or to explain individual differences in school achievement.   The ways in which people assess their intellectual abilities affect their motivation.  It's possible that these perceptions also affect individual differences in knowledge acquisition and in academic achievement.

Likewise, personality can be measured with tests.  James’ conditional reasoning test (CRT) presents five alternative multiple-choice reading comprehension problems with two correct answers. The two correct answers reflect different world views, which are presumed to be revealed by one’s selection.  There are personality tests that measure the tendency to take risks.  The willingness to exert effort and to persevere can also be measured with simple tests (like processing speed tests).

We see that personality and intelligence are intertwined.  Our challenge, argue Kyllonen and Kell, is to distinguish between the construct we wish to measure (personality or intelligence) and its measurement method.   The present state of affairs, in which the construct and the measurement method are indistinguishable, lead to distortions in our perception of the constructs and in our ability to measure them properly.  Boring said that intelligence is "what intelligence tests measure".  This is also true for personality.  The desired state of affairs is to measure each of these constructs in versatile ways:  with tests, questionnaires and other approaches.  In the existing state of affairs, the method of measurement determines whether we measure personality or intelligence…

Cognitive influences on personality and personality influences on cognition

It's possible to discern between general and specific non-cognitive factors that affect performance in cognitive tests:

General factors: personality traits, attitudes, emotional reactions, habits generally operating in situations like the test situation.  Other influencing factors can be health, motivation, mood, and the person's level of attention.  Personality factors that affect performance in cognitive tests can be, for instance, openness to new experiences, flexibility, the ability to tolerate ambiguousness, frustration and difficulties and the ability to monitor performance.  Personality factors related to externalization can underlie a tendency for speed over acuity.  Anxiety can disrupt cognitive functioning.  A person can have a tendency for internal attribution (I'm not smart enough/when I make an effort I usually succeed) or a tendency for external attribution (these questions are dumb/ I was lucky to be asked about things I just learned in class).  A person's performance can be affected by his perception of his intelligence/cognitive abilities as fixed or flexible.

Specific factors:  attitudes, emotional reactions or habits that arise in response to a specific test.  Training in a similar task can affect the quality of response to a specific test.  Relevant background knowledge can affect the quality of response to a specific test.  The health and fatigue of the child at the time of testing can also affect performance.  If the child is troubled at the time of testing – this can impede his ability to concentrate.

On the other hand there are cognitive and attentional factors that affect performance on personality questionnaires.  A child's linguistic ability can affect his reading comprehension (or listening comprehension, in case we read the items for him).  A child's ability to understand complexity in the wording of the items (fluid ability) and his ability for introspection and mentalization (in its cognitive aspects) can also affect performance.

A child's ability to observe himself from a third person viewpoint, the attentional capacities of the child, the test being culturally and linguistically appropriate for the country in which it is being used, the child's experience or familiarity with questionnaires, the child's strategies for dealing with questionnaires and/or his ability to form such strategies – all these factors influence performance.

In principle, test developers want to minimize the impact of non-cognitive factors on performance in cognitive tests and the impact of cognitive factors on personality questionnaires.  On the other hand, the existence of such influences emphasizes the extent to which these dimensions are not separate in reality.

Typical versus maximal performance.

Personality traits are often defined in terms of typicality—stable patterns of behavior over an extended period of time. If person A frequently acts in an assertive, talkative manner across a wide variety of everyday situations, she would be considered more extraverted overall than person B, who is only moderately talkative and assertive on average. However, person B, if properly motivated, may be able to act in ways more extraverted than usual, and the upper limit of person B’s extraversion may even exceed person A’s, because of situational press.

Intelligence is usually conceptualized and measured in terms of maximal performance - as what people are able to do.  Intelligence is defined as the limit of a person’s intellectual repertoire, which can be expressed when that person is exerting maximum effort.  When intelligence tests are administered under high-stakes conditions, all individuals are expected to be maximally motivated and, as a consequence, cognitive ability is assumed to be the primary (and perhaps only) source of test score variance. As we know from our work with children, the assumption of maximal performance does not always apply.  Nevertheless, since taking an IQ test is a limited-time event, the assumption is that a person can perform maximally throughout the test.  Maximal performance is not possible over longer time periods.

We've noted that personality traits are often conceptualized in terms of typical performance.  Personality is usually measured with questionnaires that refer to typical behaviors and thoughts, and in this respect they are typical performance tests.  But questionnaires are not tests…

What distinguishes between tests that measure maximal versus typical performance?

Maximal performance tests	Typical performance tests
Tests that measure mostly maximal performance:  Wechsler, Kaufman, REY – AVLT, BENDER.	Tests that measure mostly typical performance:  TAT, HTP, reading comprehension (as assessed dynamically with a text).
Typical items: what is South Korea's currency?  Scan these drawings as fast as you can, and when you see a ball, cross it out.	Typical items: tell me a story about this picture; draw a person.  Tell me the story you've just read.
Mostly assess an ability.	Mostly assess a tendency.
Predict ability in situations that are similar to the test situation.	Predict the ability to organize and to respond in ambiguous situations.
Expose mainly product and only a little of the process that led up to it.  Expose knowledge but not the use of knowledge.	Expose both process and product.
The problem is explicit and clear.	Very little instructions.  The person has to decipher the situation, to recognize that there is a problem and to create a solution.
The presented problems are relatively simple.	The presented problems are complex and require the synthesis of ideas, the organization of a sequence of actions, monitoring performance etc.
There usually is only one right solution.	Many "right" answers.
There is a clear performance standard according to which the response is judged.	General criteria for judging the response (coherence, logic, relation with the stimulus etc.)
Coping time with each problem is short.  Because of the short time period, it's possible to invest effort for maximal performance.	Require coping for a long time period - it's harder to maintain maximal performance for an extended time period.

Is it possible to measure intelligence with typical performance tests? Is it meaningful to talk about maximal performance in personality tests?

Typical performance tests assess the tendency to think in different life situations.  The tendency to think is affected by the person's sensitivity to identify moments that call for thinking and by his tendency to invest the necessary energy.  There are individual differences in people's tendency to look for new information in their surroundings, and to act upon the information they discover.

It's important to know how a person usually thinks in ordinary life situations, not only whether he is capable of thinking under maximal performance conditions, that call for solutions to clearly defined problems.  This distinction is important both for task with social – emotional content and for tasks with a cognitive/ achievement content.

Chamorro-Premuzic & Furhnam suggest the term "intellectual competence" as a way to broaden the traditional concept of intelligence.  Intellectual competence refers to a person's ability to acquire knowledge throughout life, an ability that depends not only on traditional cognitive abilities but also on his appraisal of his intelligence and personality traits. 

Intellectual competence is a marker for a person's ability to succeed in professional and learning environments, especially in environments that require both cognitive and emotional adaptation.

Chamorro-Premuzic, T., & Furhnam, A. (2006). Intellectual competence and the intelligent personality: A third way in differential psychology. Review of General Psychology, 10(3), 251.

Kyllonen,C &  Kell, H. (2018).  Ability Tests Measure Personality, Personality Tests Measure Ability: Disentangling Construct and Method in Evaluating the Relationship between Personality and Ability. Journal of Intelligence.  6, 32.

What predicts achievement better: g or broad abilities?

McGill, R. J., & Busse, R. T. (2015). Incremental validity of the WJ III COG: Limited predictive effects beyond the GIA-E. School Psychology Quarterly, 30(3), 353.‏  https://pdfs.semanticscholar.org/f5b5/d70077a1b7747a31bbcd5fb7b7dfcc38c2a3.pdf

What predicts achievement better:  g or broad abilities?

Well, it depends on the model of intelligence that you're using, and on the way the statistical analysis was done in the research on which you base your conclusion.

The WJ III COG examiner manual encourages primary interpretation at the broad ability level (e.g., CHC-related cluster scores). Because linking performance in reading/writing/math to the state of the child's cognitive abilities is a major use of intelligence tests, examining relationships between WJ III COG cluster scores and external achievement measures is important. These examinations are also critically important for evaluating the tenability of several models that have been proposed for use in the identification of specific learning disabilities (SLD) in children and adolescents. These and similar models utilize lower-order scores, such as the WJ III COG broad ability clusters, as a critical component for determining whether or not an individual has a learning disability.

According to Flanagan's model, a child is learning disabled if: A.  he has significantly poor reading/writing/math achievement.  B.  he has one (or two) significantly low broad ability scores.  C.  the low broad ability scores can explain the child's poor performance in reading/writing/math.  D.  the child's other broad abilities are average or above average.  E.  excluding factors (like insufficient or inappropriate instruction or emotional problems) are not better explanations of the poor reading/writing/math achievement.

Shortly after the publication of the WJ III COG, McGrew and his colleagues utilized multiple regression to examine predictive relationships between WJ III COG CHC clusters and standardized reading, writing and math measures. Their analyses provided evidence for differential predictive effects across the age span for specific CHC clusters.  I'd reviewed one of McGrew's studies in a prior post.  Here are slides from that post (click to enlarge). 

These slides contain valuable information that can help us plan the diagnostic process, identify the child's difficulties and plan an intervention.

But  McGill and Busse argue that these studies did not control for potential effects of the common variance shared by mental measures.  The common variance is a manifestation of g.  Each subtest and broad ability measures both g and the specific thing it is supposed to measure.  If a broad ability is "saturated" with g, it is a good measure of g but a poorer measure of the unique construct it's supposed to measure.

  

To investigate the tenability of the recommendation for practitioners to interpret primarily at the broad ability level, it is necessary to examine the incremental predictive validity provided by the broad abilities after controlling for the effects of variance already accounted for by the full scale score. That is, the extent to which broad abilities add meaningful information beyond what we get from the full scale IQ score.

The participants in this study were children and adolescents ages 6–0 to 18–11 (n = 4,722) drawn from the standardization sample for the WJ III COG and the WJ III ACH.  The WJ III COG measures 7 broad abilities: Comprehension Knowledge, Fluid Reasoning, Long Term Storage and Retrieval, Short Term Memory, Visual Processing, Auditory Processing and Processing Speed. The General Intellectual Ability – Extended (GIA-E) score is composed of 14 subtests, 2 subtests for each broad ability.

McGill and Busse used Hierarchical multiple regression analysis to analyze the data.   In this procedure, the full scale score is entered first into a regression equation followed by the lower order factor or cluster scores to predict a criterion achievement variable. This entry technique allows for the predictive effects of the cluster scores to be assessed while controlling for the effects of the full scale score.

The authors found that GIA-E accounted for statistically significant portions of each of the WJ III ACH cluster scores: Broad Reading, Basic Reading, Reading Comprehension, Broad Mathematics, Math Calculation, Math Reasoning, Broad Written Language, Basic Writing, Written Expression, Oral Expression, and Listening Comprehension (I'm not sure oral expression and listening comprehension are really achievement areas.  I think they belong more to the comprehension knowledge cluster and they are also affected by fluid reasoning).  The GIA-E accounted for 29% (Math Calculation Skills) to 56% (Listening Comprehension; Mdn 46%) of the criterion variable variance. 

CHC clusters entered jointly into the second block of the regression equations accounted for 2% (Math Reasoning) to 23% (Oral Expression; Mdn 5%) of the incremental variance.   The 23% incremental variance in Oral Expression was predicted by the Comprehension Knowledge cluster.  But even this unique outcome is dubious:  oral expression tests measure linguistic competency and vocabulary, that is – Comprehension Knowledge.  So it's hardly surprising that Comprehension Knowledge predicts Comprehension Knowledge…None of the other broad abilities accounted for more than 5% of achievement variance beyond GIA-E.   Although the CHC clusters contributed significant portions of incremental achievement variance beyond the effects of the GIA-E, effect size estimates were negligible.  The results from the current study indicate that practitioners who interpret CHC cluster scores on the WJ III COG, without accounting for the effects of the GIA-E risk overestimating the predictive effects of various CHC-related abilities

These results are fairly consistent with those that have been obtained from other cognitive measures like the Wechsler test.  

But how does all this fit with McGrew's results presented in the slides above?

McGill and Busse write that reverse entry of the independent variables, in this case entering the broad ability clusters first, would result in the clusters accounting for approximately the same variance proportions that were attributed to the GIA-E in this study. Consequently, the GIA-E would provide little incremental prediction. Order of entry arbitrarily determines whether scores such as the GIA mean everything or nothing.  However, order of entry is not an arbitrary process and must be determined a priori according to expected theoretical relationships between the variables and causal priority. Contemporary intelligence theory (e.g., CHC) and the WJ III COG structural model support entering the GIA-E before the clusters because the cluster scores are both theoretically and statistically subordinate to the GIA-E. Reverse entry conflicts with existing intelligence theory and violates the scientific law of parsimony (if you can predict something with one variable with the same level of success as with many variables, prefer the one over the many).

The CHC model is an integration between the Cattell and Horn model and Carroll's model.  Cattell and Horn presented a model of intelligence that included broad abilities but did not include g.  Carroll presented a model that included both g and broad abilities. To this day, many questions remain as to whether g reflects an actual latent ability or is merely a statistical artifact resulting from the tendency for all tests of mental ability to be positively correlated.

So, if I believe there's no g (like Cattell and Horn), the first thing I'll enter into the regression equation would be the broad abilities, and the results I'd get would be that the broad abilities predict achievement pretty well.  On the other hand, If I believe in g (like Carroll), I'll enter g first and find that it's the best predictor of achievement and that broad abilities do not add any meaningful incremental value…

What's more, it's obvious that FSIQ predicts achievement.  Obviously, it predicts achievement better than any broad ability, since it embodies the influence of all broad abilities.  But this is not the interesting question.  What we want to know is to what extent do the components of g, that is the broad abilities, predict achievement.  I think this is a good theoretical reason to enter broad abilities first into the regression equation whether we believe in g or not.  We don't want to use g to predict achievement because it's too broad and doesn't lead to meaningful interventions.

McGill and Busse point out that incremental validity researchers have largely relied on archived standardization data to assess the predictive effects of cognitive test scores. This is problematic given that the two incremental validity studies that have been conducted using data obtained from clinical samples have found significantly diminished effects associated with the general factor with greater portions of achievement variance accounted for by factor-level scores.

Additionally whereas the cognitive variables consistently accounted for large portions of achievement variance, approximately half of the variance in the WJ III ACH variables was left unpredicted in this study. What explains this additional variance?  Maybe noncognitive variables like motivation or effort.

Does IQ predict future achievement in reading, writing and math?

Watkins, M. W., Lei, P. W., & Canivez, G. L. (2007). Psychometric intelligence and achievement: A cross-lagged panel analysis. Intelligence, 35(1), 59-68.‏  http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.397.3155&rep=rep1&type=pdf

https://pdfs.semanticscholar.org/4bd0/28fb097c364b5572ac0680ea92ff0f0bf558.pdf

http://edpsychassociates.com/Papers/IQachieveCrossLag(2007).pdf

 Fletcher and Miciak (2017) claim that "there is substantial evidence showing little difference between IQ-discrepant and low achieving children in achievement, behavior, or cognitive skills, prognosis, intervention outcomes, and neuroimaging markers of brain function".

IQ-DISCREPANT are children who have a discrepancy between their IQ score and their reading/writing/ math scores. The term IQ-DISCREPANT usually refers to children with poor reading/writing/math and (at least) average intelligence.  LOW ACHIEVING in Fletcher and Miciak's paper refers to children who have poor reading/writing/math and lower than average IQ scores.   These children do not have a discrepancy between their IQ and achievement scores.

Assertions like Fletcher and Miciak's were one of the reasons for abandoning the discrepancy definition of learning disability (learning disability as a discrepancy

between at least average IQ and poor reading/writing/math that cannot be explained by exclusionary factors) in DSM5.

This article by Watkins, Lei  & Canivez presents a slightly different picture:

In current usage, intelligence tests are thought to measure general reasoning skills that are predictive of academic achievement. Indeed, concurrent IQ–achievement correlations are substantial and, consequently, comparisons of IQ and achievement scores constitute one of the primary methods of diagnosing learning disabilities (at least when this paper was written). However, intelligence tests often contain items

 or tasks that appear to access information that is taught in school (i.e., vocabulary, arithmetic) and there has been considerable debate regarding the separateness or

 distinctiveness of intelligence and academic achievement.  This apparent overlap in test coverage, among other factors, has led some to view intelligence and achievement as identical constructs. Some researchers have suggested that the relationship between intelligence test scores and educational achievement is reciprocal, mutually influencing each other.  According to this approach, children who read a lot develop their cognitive abilities and intelligence.  Children who do not read because of learning disabilities have less opportunity to develop these abilities.  Subsequently, special education researchers have suggested that only achievement tests should be used to identify children with learning  disabilities (as Fletcher suggests).  Other researchers assert that intelligence is causally related to achievement.

In order to determine whether and to what extent IQ affects achievement (or vice versa), children must be tested twice with an IQ test and twice with achievement tests over a period of a few years. If IQ affects achievement and causes it, the correlation between the IQ scores obtained in the first measurement (IQ 1) and the achievement scores obtained in the second measurement (achievement 2) should be higher  than the correlation between the achievement scores obtained in the first measurement (achievement 1) and the  IQ scores obtained in the second easurement (IQ2).

Two thousand school psychologists were randomly selected from the National Association of School Psychologists membership roster and invited via mail to participate in this study by providing test scores and demographic data obtained

 from recent special education triennial reevaluations. Data were voluntarily submitted on 667 cases by 145 school psychologists from 33 states. Of these cases, 289 contained scores for the requisite eight WISC-III and four academic

achievement subtests.

Special education diagnosis upon initial evaluation included 68.2% learning disability, 8.0% emotional disability and 8.0% mental retardation.  The rest of the students received other diagnoses.  The mean age of students at first testing was 9.25 years and the mean age of students at second testing was 12.08.

Contemporary versions of the Woodcock–Johnson Tests of Achievement, Wechsler Individual Achievement Test, and Kaufman Test of Educational Achievement were used in more than 90% of the cases. In reading, all achievement tests included separate basic word reading and reading comprehension subtests. In math, separate calculation and reasoning subtests were available for all academic achievement instruments

Here are some interesting correlations I found in the second testing (which took place when the child had already spent about three years in special education):

Basic reading skills were correlated 0.56 with Information, 0.42 with Similarities, 0.49 with Vocabulary.

Reading comprehension was correlated 0.64 with Information, 0.54 with Similarities, 0.47 with Picture Arrangement, 0.50 with Block Design, 0.60 with Vocabulary, 0.50 with Comprehension subtest.

Mathematical calculations were correlated 0.62 with Information, 0.55 with Similarities, 0.52 with Picture  Arrangement, 0.53 with Block Design, 0.57 with Vocabulary and 0.55 with Comprehension.

Mathematical reasoning was correlated 0.70 with Information, 0.63 with Similarities, 0.52 with Picture Arrangement, 0.58 with Block Design, 0.67 with Vocabulary, 0.65 with Comprehension.

The relatively high correlation of Information and Vocabulary with all achievement tests stands out.

In the first testing (before the child entered special education) the highest correlations were found between those same IQ subtests and achievement tests, but correlations were generally lower. The reason for this is unclear to me and the   researchers do not explain it.

Another thing that stood out to me was that the mean of the group of children in the Verbal Comprehension and Perceptual Organization indices did not change between the first and the second testing. This may be an indication of the stability of intelligence.  On the other hand, this may mean that the intervention the children may have received in special education did not improve their crystallized knowledge.

Even more striking is the fact that the average scores in basic reading, reading comprehension, mathematical calculations, and mathematical reasoning have not changed during these two years and eight months. This means that the children did

not make progress in their skill level relative to the norm, but on the other hand, they also did not fall behind. Another interesting thing is that the children's average scores in the achievement domains were average (around 85), not lower.

Oh, Glutting, Watkins, Youngstrom, and McDermott (2004) demonstrated that both g (general intelligence) and Verbal Comprehension contributed to the prediction of academic achievement, although g was at least three times more  important than Verbal Comprehension.

In the present study, the average correlation between IQ1 and Achievement2 was 0.466 while the average correlation between Achievement1 and IQ2 was 0.398. This means that IQ predicts achievement, not the other way around.

IQ tests were built by Alfred Binet to measure (and predict) the ability of students to succeed at school. This basic feature of IQ tests has been empirically supported for more than 100 years and is also supported by this study.

The assertion that IQ predicts future achievement has been tested with students in regular education.  In this study, it was examined with special education students and has also been confirmed. Some researchers have suggested that correlations between reading and IQ tests may often be an artifact of language, which affects both reading and intelligence.  By this line of thinking, reading difficulties lower

 IQ scores over time, and cause them to be weak predictors of achievement in students with learning disabilities.

One of the most influential researchers in the field of reading, Linda Siegel, wrote in 1998: “low scores on the IQ tests are a consequence, not a cause, of … reading disability”. I can find some logic in this argument, but I would mitigate it and say:

poor scores in some IQ subtests may also be caused by learning disability.

 However, this position was not confirmed by the present results nor by those of Kline, Graham, and Lachar (1993), who found IQ scores to have comparable external validity for students of varying reading skill. Nor was such a conceptualization supported by the relatively high long-term stability of WISC-III IQ scores among more than 1000 students with disabilities. Further, IQ has been a protective factor in several studies. In a longitudinal analysis, Shaywitz et al. (2003) found that two groups of impaired readers began school with similar reading skills and socioeconomic characteristics, but those students with higher cognitive ability became significantly better readers as young adults. A meta-analysis of intervention research for adolescents with LD demonstrated that IQ exercised similar protective effects (Swanson, 2001). A New Zealand 25-year longitudinal study found strong relationships between IQ at age 7 and 8 and academic achievement at ages 18– 25 years, independent of childhood conduct problems as well as family and social circumstances (Fergusson, Horwood, & Ridder, 2005). In sum, considerable evidence contradicts the assertion that IQ has no predictive or seminal relationship with academic achievement.

 The present study provides evidence that psychometric intelligence is predictive of future achievement whereas achievement is not predictive of future psychometric intelligence.

In conclusion, Fletcher and Miciak argue that there is no difference between children with and without an IQ-Achievement discrepancy in achievement, behavior,

 cognitive abilities, prognosis, intervention outcomes, and neuroimaging markers of brain function.

This study suggests that there is a difference in prognosis between these two groups of children.