Reconstruction of long-term complex system dynamics from short-term fragments of observational records using high-order Kalman filtering

Complex system dynamics and structures can often be evaluated through their reconstructions from available fragmented samples, which may be relatively short, with paleoclimatic and other geophysical time series obtained from proxies such as tree-ring data, as well as DNA and protein structures obtained from short reads among prominent examples. Although existing reconstruction methods focus on capturing short-term dynamics, they largely overlook long-term evolution, failing to account for the persistent long-range correlations inherent in these systems. To address this gap, we propose a generalized approach that unifies Kalman-based filtering with Kasdin’s method for the generation of colored noise. We validate its performance using both simulated time series and fragmented tree-ring width records drawn from a dendrochronological archive. The advantage of the proposed technique is that it does not require any detailed knowledge of the data properties beyond the Hurst exponent at the time of application. As a single free parameter, the Hurst exponent remains stable and representative for a given class of records, offering a powerful and accessible tool for an adequate reconstruction of the long-term behavior of complex systems.