DOI: 10.3390/su182111300 ISSN: 2071-1050

When Pilots Overlap: Contamination Bias in Difference-in-Differences Evaluations of China’s City Pilot Policies for Sustainable Development

Chao Cheng, Jinlin Zhang

China tests sustainability policies, from low-carbon and sponge cities to new-energy-vehicle promotion, in city pilot programmes, and their effects are usually estimated one programme at a time with staggered difference-in-differences (DID) designs. The programmes, however, overlap. Of 297 cities, 169 hosted two or more of ten national programmes by 2022, and 75–95% of each programme’s treated city-years overlapped with another. Monte Carlo simulations calibrated to city PM2.5, with known effects for every programme, show that this overlap contaminates DID evaluations. When each co-located programme is as effective as the focal one, two-way fixed effects, event-study and heterogeneity-robust estimators overstate the focal effect by a median 78–139%. A programme with no effect is found significant in 11–22% of simulated evaluations when co-located effects are small (0.6%), and in 71–100% in a stress test that sets each to the median of three verifiable published estimates (−3.45%). Adding dummies for the other programmes, the usual robustness check, reverses the bias, and pretrend tests rarely detect the problem. Because the contamination is a weighted sum of the other programmes’ effects, with weights fixed by the treatment pattern, these weights can be computed before outcome data are examined; we provide code and a checklist for evaluations of sustainability policies.

More from our Archive