Deep Semantics Analysis for Scale Development: Replicating Psychometrics Validation Process with Large Language Models

The present study examines whether large language model (LLM) embeddings can approximate the psychometric validation process of a newly developed psychological scale. Using the Academic Psychological Distress Scale (APDS) as a case study, we replicated the full validation pipeline, traditionally performed on human response data, by applying exploratory and confirmatory factor analyses (EFA and CFA) to semantic similarity matrices derived from LLM embeddings. We first computed cosine similarity among all 58 APDS items and found a moderately strong correspondence with the participants’ correlation matrix (ρ = 0.57). An EFA performed on the embedding-based similarity matrix yielded a six-factor solution explaining 79.1% of the variance. The best-performing model (32 items, loading cutoff = 0.70) demonstrated acceptable model fit when tested on participants’ responses (CFI = 0.88; TLI = 0.87; RMSEA = 0.07–0.08; SRMR = 0.06) numerically comparable to the fit obtained in the original human-based validation based on 367 participants’ response. Reliability estimates were satisfactory for most factors (ω/α = 0.82–0.95), although one factor showed lower internal consistency. The percentage of overlap between human-derived and embedding-derived structures, indicated an overall structural correspondence of 64%, with factor-level overlap ranging from 50% to 100%. While embeddings reproduced most dimensions of academic distress, they failed to recover the demoralization factor and instead introduced an additional behavioral cluster. These findings suggest that LLM embeddings can capture substantial aspects of the latent structure of psychological constructs and can support early-stage, low-data, embedding-first scale development. We propose a linguistic validity pipeline as a methodological framework for integrating semantic embeddings into psychometric validation procedures, highlighting their promise as a complementary tool for human-centered assessment.