Computational text indicators of epistemic norms in higher education: auditable cross-validated prediction against a weak observed household-resource baseline
Eduardo Antonio Bautista, Carlos Alberto Echeverría Mayorga, José Miguel EsperanzaPurpose
This study tests whether computational text indicators of epistemic norms add out-of-sample predictive value for normalized academic results beyond a weak baseline of directly observed household/resource indicators.
Design/methodology/approach
Data comprise 263 students observed in two waves across five sites. Prediction uses 263 follow-up records. The baseline contains seven observed indicators. Archived epistemic counts were generated through pre-assigned mappings for closed items and LDA/topic-model-assisted processing for open responses. A deterministic 10-fold split (seed 60126) evaluated raw counts, within-fold PCA, and fixed outcome-independent ENA coordinates using pooled OOF R2, RMSE and MAE.
Findings
No epistemic specification reliably improved the baseline. Pooled OOF R2 was −0.009 (M0), −0.065 (M1), −0.032 (M1-PCA) and −0.008 (M2). M1 significantly worsened MAE after Holm correction; H1 and H2 were not supported.
Research limitations/implications
The baseline is weak, no prior-achievement variable is available, corpus-level reliability of automated epistemic variables is unknown, ENA coordinates were fixed before resampling, and program representation is imbalanced.
Practical implications
The indicators should not be used for high-stakes decisions without prospective measurement, subgroup-fairness and instructional-utility validation.
Originality/value
The study provides an auditable reporting object linking feature provenance, leakage risk, resampling controls and reporting requirements, and shows why null or adverse predictive results should be retained.