DTLI: Distribution transformation-based lightweight learned indexing for data lakehouse query optimization

With the exponential growth of data scale, modern analytical platforms such as data lakehouses face severe query performance bottlenecks. To address the limitation of traditional indexes and existing learned indexes in balancing high-efficiency queries with lightweight structures, this paper proposes DTLI, a lightweight learned index architecture based on distribution transformation. We first systematically evaluate the distribution transformation performance of three generative models, Variational Autoencoder (VAE), Normalizing Flow, and Diffusion Model, and select the optimal Block Neural Autoregressive Flow (B-NAF) as the transformation operator to map original complex distributions to near-uniform distributions. On this basis, we propose a changepoint-based piecewise fitting algorithm for cumulative distribution functions, constructing a minimal binary tree index structure containing only model nodes and pointer nodes to achieve index lightweighting. Finally, we design local and global indexing strategies adapted to the partitioning characteristics of data lakehouses, completing the integration of DTLI on the Apache Hudi platform. Experiments demonstrate that DTLI significantly outperforms native indexes and learned indexes including RMI, PGM, and NFL across multiple datasets. Specifically, DTLI improves average throughput by 72.55%, 50.87%, 38.60%, and 5.21% over B+Tree, RMI, PGM, and NFL, respectively, while reducing 99th percentile tail latency by 57.52%, 56.44%, 29.35%, and 15.81%, with advantages amplifying as data scale increases. Ablation studies confirm that distribution transformation can improve the throughput of existing indexes by over 38%. Currently, DTLI is designed and evaluated for one-dimensional numerical keys. Its extension to multi-dimensional or non-numerical data is non-trivial and remains as future work. Additionally, the distribution transformation relies on offline training, and incremental update mechanisms for dynamic scenarios require further investigation.