Biobank-scale self-supervised genotype pretraining yields transferable representations for human genetics

Abstract

Biobank-scale genotyping has created unprecedented cohorts for human genetics, yet the field still lacks scalable methods for learning transferable representations from largely unlabeled individual-level genotype data. Here, we introduce SNPBERT, a self-supervised transformer pretrained on genotypes from 485 923 UK Biobank participants using masked rsID prediction. SNPBERT embeddings recovered known population structure across UK Biobank and three external datasets, performed comparably to principal component analysis, and supported the identification of ancestry-informative marker SNPs. Fine-tuning with an attention-based multiple-instance head generated disease-specific risk scores that provided complementary information beyond conventional polygenic risk scores. The relative increase in Nagelkerke’s R2 ranged from 0.7% to 3.3%, and independent associations were observed for seven of the eight chronic diseases examined. In celiac disease, attention prioritization recapitulated major histocompatibility complex signals while elevating variants in CCHCR1 and NOTCH4 relative to genome-wide association study (GWAS) ranking. These results support biobank-scale genotype pretraining as an early step toward foundation modeling for individual-level genotype data.