LTVAE : Hybrid LSTM –Transformer VAE Architecture for Molecular SMILES

ABSTRACT

A hybrid LSTM–Transformer Variational Autoencoder (LTVAE) framework is introduced in this paper for learning compact and chemically meaningful latent representations from SMILES strings, providing a SMILES‐to‐latent‐to‐SMILES representation model trained on ZINC15 and evaluated on ZINC15 and an out‐of‐distribution dye dataset. In the proposed model, sequential and structural dependencies in molecular strings are encoded using a bidirectional LSTM, while a Transformer‐based decoder is employed to reconstruct valid molecular structures in the evaluated settings. When trained on a curated ZINC15 dataset, the framework achieves reasonable reconstruction performance, with 98.58% token‐level accuracy, 92.19% string‐level accuracy, a 99.30% chemical validity ratio, and an average Levenshtein distance of 0.29 using a 64‐dimensional latent space. Reconstruction under an out‐of‐distribution evaluation setting is evaluated on a structurally distinct dye dataset, where 95.56% token‐level accuracy and 85.30% validity are achieved. The learned latent space preserves trends aligned with both general molecular descriptors and dye‐relevant structural descriptors, particularly conjugation length and aromatic surface fraction, with Fsp providing complementary information on molecular saturation. Controlled perturbations in the latent space suggest smooth and chemically consistent structural variations, illustrating the continuity and interpretability of the learned latent manifold.