Context Sufficiency in Transformers: Information-Theoretic Limits, Native Functional Dependence, Representational Accessibility, and Predictive Readout

We present a unified experimental decomposition of transformer prediction that separates information availability, native functional dependence, representational accessibility, and downstream predictive readout. For finite-vocabulary order-k Markov sources, the context penalty is exactly HL−H*=I(Xt+1;St∣Ct(L)) and vanishes for L≥k; all 15 sampled dense kernels show a strictly positive Hk−1−H* gap. Scratch transformers approach the corresponding context-limited optimum but retain optimization-dependent residuals. Post-training state-changing interventions steer the native predictive distribution toward the exact counterfactual oracle, and source-supported alternatives preserve 0.72–0.77-nat tracking gains while matched out-of-state controls remain near zero. In frozen Qwen2.5 representations, repeating the analysis over ten independent vocabulary maps per fixed checkpoint preserves the sharp state accessibility transition at L=k; Pythia-410M provides a cross-family replication. Capacity and sample-size controls show that residual predictive readout loss can arise even when the true state is supplied directly. Finally, a controlled hidden Markov model extension replaces the visible suffix state by a latent Bayesian belief state: exact filtering, scratch prediction, posterior probing, and predictive readout remain separately measurable, while the predictive gap to a 256-observation Bayesian reference decays smoothly rather than at a finite boundary. The results support a bounded four-level account of context sufficiency without equating decodability with native causal use.