An Information Fusion Based Decision‐Making Framework for Generative Performance Assessment of Large Language Models

ABSTRACT

A subjective‐objective information ‐ based decision ‐ making fusion framework is proposed to assess the generative performance of large language models (LLMs) by combining objective evidence and expert‐based subjective judgement. The Analytic Hierarchy Process (AHP) is first used to quantify the objective weights of sub‐indicators and compute the objective performance based on the standardised benchmarks. The Best‐Worst Method is then used to quantify the subjective weights of sub‐indicators based on domain experts. After that, an iterative solution method is introduced to align the divergence domain expert opinion, update the subjective weights, and compute the subjective performance of LLMs. Finally, the performance of LLM can be quantified by the weighted sum of subjective and objective performance. Comparative and sensitivity analyses are carried out to validate the feasibility and superiority of the proposed method. Overall, the proposed method can deal with the potential subjective uncertainty in expert opinion and supporting final decision‐making.