Assessing EFL Learners’ Phraseological Competence Through Association Measures: The Role of Reference Corpus Selection

ABSTRACT

Association measures, such as pointwise mutual information (PMI), have been widely used to explore learners’ phraseological competence. Yet, the validity of PMI‐based evaluations critically depends on the reference corpus employed, and the extent to which this methodological choice affects assessment outcomes has not been systematically examined. This study investigates how reference corpus selection influences PMI scores and, consequently, the evaluation of learners’ phraseological competence. We calculated PMI scores for amod , dobj , and advmod dependency pairs from the oral performances of 98 test‐takers, comparing the outcomes when using four reference corpora: the full COCA corpus (COCA‐ALL) and its specialized academic (COCA‐ACAD), news (COCA‐NEWS), and spoken (COCA‐SP) sub‐corpora. Results revealed that, first, for the same set of dependency pairs, the choice of reference corpus led to statistically significant differences in their calculated PMI scores for all three dependency types ( p < 0.001). This effect was most pronounced for dobj relations, where 88.9% of pairs exhibited substantial PMI differences across certain reference corpora. Second, this systematic bias translated directly into unreliable learner assessments. COCA‐ALL consistently assigned significantly lower median PMI scores ( p < 0.001), leading to unstable proficiency rankings (correlation coefficient dropping to 0.59). For dobj relations, this caused over half of learners (55.6%) to shift by 10 or more rank positions. These findings demonstrate that reference corpus selection is a critical factor that can systematically affect and undermine the validity of phraseological assessments.