Finite-Resolution Information from Collision Statistics

Collision statistics provide a finite-resolution view of information by measuring how often independent samples fall on the same state and form the basis of integer-order Rényi entropies. Here, we use low-order Rényi entropies to characterize finite-resolution approximations to Shannon entropy and mutual information. Specifically, we determine what population information is captured by finite collision moments, we quantify how the resulting targets differ from their Shannon counterparts, and we analyze how accurately they can be estimated from finite samples. We use the interpolation remainder to identify structural approximation error induced by extrapolating from integer-order Rényi entropies to the Shannon point. We separate this deterministic error from finite-sample estimation error: increasing sample size improves estimation of a finite-resolution target but does not eliminate its deterministic difference from Shannon entropy or mutual information. Finally, we show that finite collision moments do not generally identify Shannon entropy, and that increasing collision order shifts sensitivity toward high-probability events. Our numerical experiments illustrate the approximation–estimation trade-off and evaluate collision-based approximations alongside plug-in and Miller–Madow estimators. Together, these results provide a principled way to use low-order coincidence structure as finite-resolution information, while making explicit what finite collision moments can and cannot reveal about Shannon entropy and mutual information.