Multi-agent generative AI for concept evaluation: consistency, knowledge integration and human alignment

ABSTRACT:

Early-stage concept evaluation is critical for selecting viable designs. This study introduces a multi-agent generative AI framework for assessing concepts across four configurations: AI with retrieval-augmented knowledge, AI without external knowledge, human experts, and a hybrid approach. The findings show that AI panels tend to produce uniform evaluation patterns, while retrieval-augmented knowledge alters rating behaviour without leading to closer alignment with human judgement. Hybrid setting achieved closest alignment, indicating AI is effective when combined with expert interpretation.