What are we really measuring? The semantic confound in psychological scales and the case for theory-driven measurement.

Abstract

The replication crisis in psychology is now widely recognized as a crisis of measurement itself, not merely of questionable research practices. Recent critical work has revealed that many psychological scales may be measuring semantic associations rather than the psychological phenomena they purport to assess. This Hypothesis and Theory article makes three contributions. First, it synthesizes diverse lines of critique-including Uher's distinction between statistical modeling and measurement, and the finding that computational language models, trained solely on word co-occurrence patterns, can predict questionnaire responses with surprising accuracy-without any access to human participants' psychological states. These critiques converge on a common diagnosis: the "semantic confound," where scales capture linguistic familiarity rather than latent psychological traits. Second, it argues that the root cause of this measurement crisis is not statistical but theoretical: the absence of explicit, falsifiable theories about the target phenomena themselves. This theory vacuum has, in practice, allowed scale development to default to an inductive, data-driven approach that conflates statistical structure with psychological structure, despite the field's long-standing theoretical commitment to construct validity (Cronbach and Meehl, 1955). Third, it outlines what theory-driven measurement would require: axiomatic assumptions, phenomenologically distinct state definitions, formal causal structures, and a mapping function from observable behaviors to latent states. We demonstrate that these conditions are neither arbitrary nor utopian by examining four research traditions-psychophysics, structured diagnostic interviews, item response theory, and cognitive diagnostic models-that each partially satisfy one or more of them. The Holographic Coding Framework (HCF), recently proposed as a theory-driven approach to consciousness measurement, is briefly introduced as an illustrative case of this alternative paradigm. We conclude by calling for a fundamental shift in quantitative psychology: from measurement models that fit data to theories that drive measurement. Standardizing poor measurement entrenches the validity crisis; theorization must precede standardization.