A multiscale deep learning framework for predicting stratified shear flow instabilities

Kelvin–Helmholtz (KH) instability in stratified shear flows exhibits strong multiscale dynamics, in which large-scale vortical structures evolve into smaller-scale motions through nonlinear interactions. Accurate prediction of KH dynamics is important for understanding mixing and transport in geophysical and environmental flows. Although high-fidelity simulations such as direct numerical simulation and large eddy simulation (LES) can resolve these dynamics, their computational cost limits their use across a wide range of flow conditions. Data-driven reduced-order models based on convolutional neural network autoencoders (CNN-AEs) and long short-term memory (LSTM) networks offer a promising alternative but often underrepresent dynamically important fine-scale structures. This study presents a parameter-conditioned multiscale deep learning framework for reduced-order prediction of stratified KH instability. The framework employs a state CNN-AE to capture dominant large-scale flow structures, a difference CNN-AE to learn finer-scale motions, and a parameter-conditioned LSTM to jointly evolve both latent representations using the gradient Richardson number (J) and the shear-to-stratification thickness ratio (R). The model is trained using vorticity and buoyancy fields from two-dimensional LES-like simulations spanning multiple flow configurations and is evaluated on both training and previously unseen test cases. The framework accurately reproduces the evolution of KH billows and preserves fine-scale structures throughout the nonlinear stages of instability development, as assessed through comparisons with the conventional CNN-AE–LSTM framework and LES using statistical metrics and physics-based diagnostics, including perturbation enstrophy, instability growth rates, and kinetic energy spectra. The proposed model also enables long-horizon forecasting from a single early-stage flow snapshot and generalizes successfully to previously unseen (J,R) combinations.