How Detectable Is Machine-Generated Text? A Preregistered Audit of Post-Training, Scale, and Task Effects

Why is machine-generated text detectable, and how much does the answer depend on the model’s scale, its post-training, and the writing task? We measure detectability with a validation-tuned linear probe on frozen embeddings from RoBERTa, a pretrained transformer language encoder, under a preregistered protocol in which every model stage receives an identical plain prompt, every output is exactly 256 tokens from a single attempt, and all evaluation rules were fixed before any score existed. Across nine model configurations evaluated in thirteen model–task cells (four Tulu-3 post-training stages under both rewriting and free generation; five Pythia sizes from 70M to 6.9B parameters), detectability at full instrument strength is near-perfect in every cell: ten of thirteen cells reach an area under the curve (AUROC) of 1.0, including the unaligned base model. With the instrument deliberately weakened to a 32-token view, exact ordered tests support a small increase in detectability across post-training stages in both tasks, but the rewrite and free-generation conditions do not differ (difference-in-differences −0.002, 95% CI [−0.033,0.029]), and no association between detectability and model scale is detected within the examined 70M–6.9B range (exact Spearman ρ=−0.1, p=0.475). At the level of individual texts, rewrites that copy more of their source are reliably harder to detect (clustered ρ=−0.27 to −0.53, all ten generators negative). The picture that emerges, for the examined configurations, domain, and instrument, is pervasive detectability with small, task-independent post-training effects, and it implies that detector evaluations need ceiling checks, matched prompts, and instrument-strength ladders before their comparisons can be trusted.