LLM-based voice chatbot surveys as an alternative to post-experience questionnaires: probe-controlled, ultra-short field interviews

ABSTRACT:

Chatbot-based surveys offer low-burden, in-situ data collection, yet unconstrained LLMs often drift from research aims. We conducted 359 ultra-short, post-experience voice interviews in a public venue to compare a framework-guided LLM, an unconstrained LLM, and fixed questions. The guided approach produced significantly longer responses than fixed questions and yielded the richest diversity of process-specific accounts. These findings show that probe control is essential for eliciting actionable, experience-grounded feedback in real-world, time-limited settings.