BackSmith: A Systematic Approach to Testing Compiler Backends

Compiler backends are critical for translating high-level code into efficient machine instructions, yet they remain relatively underexplored in compiler testing. Effective backend testing requires programs that expose low-level backend behaviors, but such features are difficult to generate and are frequently eliminated by earlier optimization passes. As a result, existing testing approaches often fail to adequately exercise backend behaviors and are therefore less effective at uncovering backend defects.

We present BackSmith, a black-box approach for testing compiler backends across compilers and architectures. BackSmith generates code snippets with two complementary properties: backend-oriented features that directly stress backend mechanisms such as instruction selection and register allocation, and optimization-resistant features that preserve program diversity by resisting excessive middle-end canonicalization. To further increase coverage of rare but critical backend behaviors, BackSmith also generates code snippets whose compiled assembly rarely arises during random generation. It then integrates all three kinds of features into seed programs for backend testing.

We evaluated BackSmith on 16 mature GCC and LLVM backends. Over five months of testing, BackSmith uncovered 104 previously unknown backend bugs, 88 of which have been confirmed or fixed, demonstrating the effectiveness of our approach in systematically exposing backend defects.