A Post-Processing Framework for Crowd Worker Responses Using Large Language Models

To develop quality crowdsourcing systems, aggregating responses from workers is a critical issue. However, it has been difficult to construct an automatic mechanism that flexibly aggregates worker responses in natural language. Accordingly, responses need to be collected in a standardized format, such as binary-choice or multiple categorizations, to avoid large aggregation costs. Recently, with the advent of large language models (LLMs), natural language responses can be automatically and flexibly aggregated. We propose a framework that uses LLMs to flexibly aggregate natural language responses from workers and, as a promising example, consider this framework for crime detection from surveillance cameras using crowdsourced cognitive abilities. In an experiment using subjective evaluation, our proposed framework is shown to be effective for automatically aggregating natural language responses from crowd workers.