Soft Non-diagonality Penalty Enables Latent Space-Level Interpretability of Parameter-Efficient Peptide LM at No Performance Cost

Abstract

Emergence of large scale protein language models (pLMs) has led to significant performance gains in predictive protein modeling. However, it comes at a high price of interpretability, and efforts to push representation learning toward explainable feature spaces remain scarce. The prevailing use of domain-agnostic and sparse encodings in such models fosters a perception that developing both parameter-efficient and generalizable models in a low-data regime is not feasible. In this work, we explore an alternative approach to develop compact models with interpretable embeddings while maintaining competitive performance. With the bidirectional long short-term memory autoencoder (BiLSTM-AE) model trained on positional property matrices, we introduce a soft weight matrix nondiagonality penalty and a one-hot encoded sequence clustering-based contrastive loss. As evidenced by Jacobian analysis, the penalty aligns embeddings with the initial feature space, whereas the contrastive loss organizes the latent space semantically. This combination leads to consistent improvements in performance on a suite of eight common peptide biological activity and physicochemical properties benchmarks. The use of amino acid physicochemical properties and density functional theory (DFT) derived cofactor interaction energies as input features provides a foundation for intrinsic interpretability, which we demonstrate on fundamental peptide properties. The resulting model is over 33,000 times more compact than the state-of-the-art pLM ProtT5. It demonstrates performance stability across diverse benchmarks without task-specific fine-tuning, showcasing that domain-tailored architectural design can yield highly parameter-efficient models with fast inference and preserved generalization capabilities.