"Like Taking the Path of Least Resistance": Exploring the Impact of LLM Interaction on the Creative Process of Programming

As AI takes on more of the generative work that once required human imagination, despite documented limitations in creative ability, a critical question emerges: How does GenAI affect users’ creativity? Through a within-subject study followed by retrospective interviews with (N=20) programmers, we investigated the impact of LLMs on participants’ process of creative thinking in programming and the creativity of generated solutions. Across two conditions (LLM-assisted vs. unassisted), participants using LLMs had significantly shorter idea-generation periods ( \(p=1.6\times 10^{-5}\) ) and fewer creative moments ( \(p=4.4\times 10^{-7}\) ). Qualitative analysis of participants’ interactions and interviews revealed four different human-LLM collaboration modes supporting various problem-solving strategies. However, a comparative analysis of the generated solutions shows that while LLMs can help generate more correct and functional code, LLM-assisted solutions to algorithmic tasks were significantly more similar to one another, even though the number of unique ideas was comparable. Based on our findings, we discuss design implications and considerations for effectively using LLMs to support user creativity.