$\begingroup$

Apologies in advance for the long answer. I tried to narrow down the scope by focusing on only a single construct, and only a single aspect of validity, and it still turned out like an essay...

Let's take intelligence research as an example. This work started with an intelligence concept – a fairly vague and ambiguous idea about a personality trait that describes a person's cognitive abilities. From this, a construct was hypothesized: A physical – but unobserved – mechanism that implements intelligence, called faculties (or g-factor). The next step was to figure out how such faculties might be measured. There was a desire to find simple tools to do this, and so a variety of IQ tests – mostly written, and some not – were developed.

It is not clear in any sense that intelligence as a (folk) concept has any ontological basis. Presumably the construct of cognitive faculties should have a physical basis, but in practice it doesn't appear to. And IQ tests do not have as their primary objective the measurement of such faculties anyway (although they may be useful indicators). IQ tests were originally developed to predict academic performance, and test validity continues to be measured against that, which is typical of psychometrics in general. So in what sense (if any) are these terms related to one another?

This is a question of validity, or more specifically, construct validity. The relationship between concept, construct, and measurement, is a dynamic and bi-directional one. An often cited example for intelligence research is the case of the Brazilian street vendors. In a classic 1988 study by Geoffrey Saxe, Brazilian street children who often work as candy vendors were compared with rural non-seller counterparts for mathematical abilities. Though typically measuring lower in IQ, the street vendors matched and in some cases outperformed their rural counterparts on a variety of practical applications involving arithmetic.

Why the discrepancy? It turns out that children with a formal education in math are good at solving mathematical problems framed in a formal manner. Lacking a school education, street kids do not acquire these skills, and so do poorly on tests requiring them to identify and work with symbols such as numbers and operators. However, working as street vendors, these children are very proficient at arithmetic involving currency, such as calculating costs and making change – without any aids. So when presented with math problems framed as currency operations – such as: how much does it cost to buy 3 of this and 2 of that; or how much change do I get back from this bill – they perform better than educated children of the same age.

Similar results were found by Jean Lave (1988): In her study, Berkeley housewives performed significantly better in mathematics framed as grocery shopping tasks - calculating discounts and coupons for example - than framed as typical classroom math problems. In another study by Ceci and Liker (1986) gambling experts were capable of complex mathematics in calculating handicaps in horse-racing, but yet had no difference in IQ than non-experts.

What can we learn from this? Many different definitions of intelligence have been considered. One interpretation of these results is that intelligence is not just a general trait as implied by the single IQ number result, but is composed of a number of independent domain-specific intelligences, some of which are missed by the test. A different interpretation is that an unbiased measure of general intelligence requires framing questions in a context familiar to each subject. One thing that seems certain is that a general factor of intelligence that carries over to any domain appears less ontologically tenable.

Most evaluations of construct validity are domain-specific – they depend on what the theory being validated purports to describe, and as indicated, that too may change based on empirical findings. Of course, changes may not be quick to materialize – the concept of intelligence, and intelligence testing, have both certainly co-evolved over the years, but the rate of change has not been commensurate with findings such as the above, and is often skewed by political agenda. So in the mean time, it is not unusual for researchers to pursue red herrings – cognitive concepts that eventually turn out to be ontologically untenable. But this is hardly unique to cognitive science, and is a natural part of the scientific process of learning.