Contextualized Code Pretraining for Code Generation

As code generation becomes increasingly central to improving software development efficiency, modern code models are largely trained and evaluated on code with natural-language descriptions. In real projects, developers often implement missing functions under limited project-specific artifacts, while the local call-site context is already available in the surrounding code. This usage context provides actionable cues about expected behavior, but existing models are not explicitly optimized to leverage it reliably, leading to implementations that may not integrate smoothly with surrounding usage in repository settings.

In this work, we propose contextualized code pretraining, an invocation-aware framework that integrates calling context into both the training and evaluation of code models. Using static analysis, we automatically extract large-scale caller–callee pairs from real repositories to construct pretraining tasks and benchmarks that condition generation on the calling context. We train

, the first code models pretrained with invocation-aware objectives spanning multiple sizes, and evaluate them on , a new benchmark featuring realistic scenarios. Experiments across our two primary benchmarks show that outperforms comparable-scale models and remains competitive with larger ones. Our 220M and 0.5B models achieve 16.58% and 22.81% pass@1, surpassing baselines on . These results highlight the importance of calling context in realistic code generation.