Towards High-Quality Neural Program Repair: Learning from Positive Code Changes

Automated Program Repair (APR) seeks to fix bugs without human intervention, aiming to free developers from the tedious and time-consuming debugging process. In recent years, with the advancement of deep learning techniques, Neural Program Repair (NPR) has emerged as a promising research direction for APR. NPR tools have shown significant progress, especially in terms of the number of bugs correctly fixed across well-established benchmarks. However, recent studies have raised concerns regarding the reliability of NPR tools, revealing that tools trained on bug fix data also perform well in fault injection tasks. This suggests that existing NPR tools may not be genuinely fixing bugs, but rather simply making code modifications. As a result, the patches generated by NPR tools have a high likelihood of introducing new bugs, posing a potential risk to software reliability.

In this study, we investigate the underlying reasons behind this phenomenon. We begin by conducting an empirical study to investigate different types of code change operations. By performing clustering analysis on a large number of code change samples, we categorize code changes into three distinct groups based on the nature of fixing code change patterns: positive, mixed, and negative. Next, we conduct experiments to investigate how different types of code changes influence NPR tools’ performance. We utilize the code change templates derived from the manual analysis of these clusters to filter the existing dataset and retrain NPR models using this refined dataset. Experimental results show that the retrained NPR models outperform their non-retrained counterparts in terms of precision, and the generated patches contain fewer negative code changes. Based on the results, we discuss insights into the limitations of previous NPR studies and hope to provide researchers with a new perspective for enhancing their reliability by improving dataset quality.